toga-ai 1.0.740 → 1.0.741

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -32,6 +32,7 @@
 | [_Model::save() vs raw _Query — no atomic conditional update](features/model-save-vs-query-atomic-update.md) | `_Model::save()` is a plain load-then-write ORM primitive and **cannot express an atomic conditional update** (an optimistic-concurrency / row-claim guard such | _underscore/Model.php, _underscore/Query.php, _underscore/Model/Rate/Subscription.php |
 | [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, _underscore/Trait/Netsuite/SalesOrder.php, worker2/Worker/Netsuite/SalesOrder.php, worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Worker/Netsuite/Opportunity.php |
 | [NetSuite Sales Order sync — ship-to address, phone, and PO reference sourcing](features/netsuite-salesorder-address-phone-sync.md) | `_Trait_Netsuite_SalesOrder` is the **shared** sales-order importer composed into **22 client models** (every client on the dbchanges2 `netsuite` module). | _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Model.php, dbchanges2/_modules/netsuite/2026-08-10a - AddressPhoneNumberApiRoleAcl.sql |
+| [Outbound third-party API rate budget (_Component_Api_RateBudget) — designed + staged, NOT merged](features/outbound-api-rate-budget.md) | A **shared outbound rate budget for every third-party API** we call from 2.0 (ClickUp, NetSuite, FedEx, UPS, …): one small component plus one `Core` table, keye | _underscore/Component/Api/RateBudget/RateBudget.php, _underscore/Component/Api/Clickup/Clickup.php, _underscore/Route.php, _underscore/Query.php, dbchanges2/Core/2026-09-01a - Add ApiRateBudget table.sql, worker2/Worker/Clickup/Api.php |
 | [Legacy page meta (Page::meta) & context-scoped ClientRecordFieldSettings](features/page-meta-context-field-settings.md) | `_Model_Core_Page::meta()` is the **legacy** page-meta resolver behind `GET /pages/meta?slug=<slug>` — still the live path for `toga2-supply` and other pre-Surf | _underscore/Model/Core/Page.php, _underscore/Model/Client/TableView.php, toga2-supply/src/components/ui/Tables/PrimaryTable/PrimaryTable.tsx, toga2-supply/src/components/ui/Tables/PrimaryTable/helperFunctions/formatTableData.tsx |
 | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/Model.php, _underscore/Query.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, api2/Controller/Index.php |
 | [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona **names** in multiple languages by adding a per-language **sidecar** table `PersonaTranslations`, reusing the platform's existing metadata-driven | _underscore/Model/Client/PersonaTranslation.php, dbchanges2/Client/2026-07-22b - PersonaTranslations.sql, dbchanges2/Core/2026-07-22a - PersonaTranslationsRecord.sql, dbchanges2/Client/2026-07-22c - PersonaTranslationsAcl.sql, dbchanges2/Client_CompassCanada/2026-07-22 - PersonaTranslationsFrench.sql, toga2-commerce/src/pages/Account/view/MySettingsView.tsx |

package/knowledge/2.0/apps/_underscore/features/outbound-api-rate-budget.md

@@ -0,0 +1,190 @@
+---
+title: Outbound third-party API rate budget (_Component_Api_RateBudget) — designed + staged, NOT merged
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: draft
+updated: 2026-09-02
+owners: [ajean]
+files:
+  - _underscore/Component/Api/RateBudget/RateBudget.php
+  - _underscore/Component/Api/Clickup/Clickup.php
+  - _underscore/Route.php
+  - _underscore/Query.php
+  - dbchanges2/Core/2026-09-01a - Add ApiRateBudget table.sql
+  - worker2/Worker/Clickup/Api.php
+related:
+  - ../architecture.md
+  - ../../worker2/features/creating-worker-actions.md
+  - ../../worker2/features/clickup-project-routing.md
+  - ./async-query-execution.md
+  - ./model-save-vs-query-atomic-update.md
+---
+
+## Summary
+
+A **shared outbound rate budget for every third-party API** we call from 2.0 (ClickUp,
+NetSuite, FedEx, UPS, …): one small component plus one `Core` table, keyed by a
+`bucketKey` string. It counts outbound calls per fixed 60-second window so we can *measure*
+our real request rate before enforcing any cap.
+
+**Status — nothing is shipped.** The component and its migration exist only as local staged
+files; they are not committed to `_underscore` or `dbchanges2`, and no PR contains them. They
+replace a rejected ClickUp-only design (533-line worker class + 106-line migration = 639 lines)
+that lives in the still-open PRs worker2 **#137** and dbchanges2 **#449**. Read this doc as a
+design record, not as live behavior.
+
+## Key files / entry points
+
+- `_underscore/Component/Api/RateBudget/RateBudget.php` — `_Component_Api_RateBudget`
+  (staged; 222 lines, ~103 lines of code).
+  - `take(string $bucketKey, int $limit): ?int` — **the entire public surface.** Increments the
+    current window's counter and returns the post-increment count, or `null` if the store is
+    unreachable. It never blocks, never sleeps, never throws on saturation.
+  - `prune()` — deletes old windows; call from an existing cron, never on the hot path.
+- `dbchanges2/Core/2026-09-01a - Add ApiRateBudget table.sql` — staged, 19 lines. Creates
+  `Core.ApiRateBudget`: `id`, `uuid`, `dtCreated`, `dtUpdated`, `bucketKey varchar(64)`,
+  `windowStart int UNSIGNED`, `requests int UNSIGNED`, with `UNIQUE KEY (bucketKey, windowStart)`.
+- Rejected predecessor (in PR #137 / #449, ClickUp-only): `worker2/Worker/Clickup/Api.php`.
+
+## How it works
+
+1. A caller about to hit a provider calls `take('clickup', $limit)`.
+2. The component upserts `(bucketKey, windowStart)` incrementing `requests`, then reads the row
+   back by primary key, and returns the count.
+3. The caller decides what to do with the number. **Counting and enforcement are deliberately
+   separate.**
+
+Design decisions worth keeping:
+
+- **One table, one class, keyed by `bucketKey`** (`'clickup'`, `'netsuite'`, `'fedex'`, `'ups'`).
+  Never a second per-provider table or class. The table lives in `Core/`, not `Team/`, because it
+  is platform infrastructure, not team tooling.
+- **Observe-only deployment is the point.** Run it for a week, read the real aggregate rate out of
+  `Core.ApiRateBudget`, then pick a threshold from measurement instead of guesswork. The
+  measurement *is* the test.
+- **Fixed 60s window, boundary computed by MySQL** (`FLOOR(UNIX_TIMESTAMP()/N)*N`) and **never by
+  PHP** — instance clock drift across the fleet would split the budget. Accepted trade-off: a burst
+  straddling a boundary is bounded at 2x nominal. If that ever matters, move to a 10s window and
+  `SUM()` the last 6 rows.
+- **Fail open.** If the store is unreachable: log once, return `null`, proceed with the call.
+  worker2 has **no DLQ and no redelivery**, so a fail-closed limiter outage would permanently
+  destroy production events — strictly worse than briefly exceeding a provider's cap.
+
+### Framework constraint: a cross-process counter needs its OWN DB alias, committed immediately
+
+This is the one piece of essential complexity carried over from the rejected design, and it
+applies to **any** shared counter or rollup in 2.0, not just this one.
+
+`_underscore/Route.php` opens a transaction on the **default** connection and holds it open until
+the job ends. So a counter written on the default alias is:
+
+1. **invisible to every other process** until the job ends — a fleet-wide total can never be read; and
+2. **a write lock held on one hot row** for up to the full 300s job budget — every worker that
+   touches the counter serialises behind it.
+
+The only escape is to register a **second connection under its own alias** and call
+`_Database::transactionCommit($alias)` immediately after the write. A "simplification" that drops
+the alias silently breaks correctness *and* adds a fleet-wide lock.
+
+### Why not the one-statement `LAST_INSERT_ID(expr)` upsert-and-read
+
+The tempting single statement —
+`... VALUES (… LAST_INSERT_ID(1)) ON DUPLICATE KEY UPDATE requests = LAST_INSERT_ID(requests + 1)`
+— is **not safe on a table with an `AUTO_INCREMENT id`**: `LAST_INSERT_ID()` returns the *new row
+id* on the INSERT path but the *assigned expression* on the UPDATE path. The 2.0 table standard
+mandates a leading `id AUTO_INCREMENT`, so the idiom would need empirical proof against the target
+MySQL 8 before use.
+
+**Decision:** upsert, then read the row back by primary key. One extra indexed lookup, obviously
+correct. The resulting read-after-write skew is **upward-only** under concurrency (another process
+may increment between our write and our read), so the number is conservative — it can cause an
+early yield but never an overrun.
+
+### Enforcement policy (once there is data) — never sleep-wait
+
+`MAX_EXECUTION_TIME_WORKER_ACTION` hard-caps any non-CronScheduler job at **300s**, and the
+JobScheduler watchdog only *writes the failure row* — it does not stop the PHP process. Sleeping to
+wait for budget therefore burns the cap and gets the job marked failed with **no retry and no DLQ**.
+See [Creating Worker Actions](../../worker2/features/creating-worker-actions.md).
+
+| Traffic class | Examples | Policy at the threshold |
+|---|---|---|
+| Bulk / sweep | sprint sync, stakeholder backfill, discovery | **Yield cleanly and reschedule the remainder.** Already chunked and resumable, so pausing costs nothing. This is where a cap gets real teeth. |
+| Event-driven / escalation | Sentry→ClickUp, webhooks, design review | **Proceed over budget, counted and alarmed.** Low volume, irreplaceable. |
+
+## Verified findings about ClickUp metering today
+
+- **The "500/min is worker2's deliberate half of ClickUp's 1000/min" premise is FALSE.** Both
+  `worker2/Worker/Clickup/Api.php` and the #449 migration assert that api2 and tools also call the
+  ClickUp API. Verified by grep across the `api2`, `tools` and `_underscore` checkouts: **zero**
+  references to `api.clickup.com` and **zero** `_Component_Api_Clickup::send()` calls in api2 or
+  tools. `tools` reads the `Team` database that worker2 populates; it does not call ClickUp.
+  worker2's realistic share is therefore **~900/min, not 500** — the limit must be re-derived with
+  the reviewer before any enforcement.
+- **~24 of ~116 ClickUp call sites would be unmetered** (measured on the merged tree via
+  `git merge-tree --write-tree` + grep): 92 metered, 6 raw `_Component_Api_Clickup::send()`, and 18
+  hand-built `api.clickup.com` endpoints — `Worker/Team/Sprint.php` (6),
+  `Worker/Netsuite/Opportunity.php` (6), `Worker/Sentry.php` (4),
+  `Worker/Netsuite/OpportunityStakeholderBackfill.php` (2). The **burstiest** loops (paginated
+  sprint sweeps, the stakeholder backfill) are among the unmetered ones, so a limiter as built
+  misses exactly the traffic most likely to breach a cap. **Call-site consolidation is the real
+  work** — any limiter is only as good as that.
+- **The rejected design's own enforcement test is blind to the biggest bypass.**
+  `TestNoDirectApiCalls()` greps only for `_Component_Api_Clickup::send(`, so it cannot see the 18
+  raw `api.clickup.com` sites, and it reports `RESULT: FAIL (2)` immediately post-merge because
+  `Worker/Monitor/Oneuptime.php` (5 unmetered calls) landed on `_production` after the branch was
+  cut. **A guard that passes while 18 bypasses exist manufactures false confidence.** This belongs
+  in CI as a ~10-line grep for `api.clickup.com` outside the component — not as a curl-invokable
+  worker action.
+
+## What was deliberately dropped from the rejected design
+
+45-vs-50 rolling-minute arithmetic, a 0.25s window-edge guard, batch reserve plus
+`register_shutdown_function` slot refunds, `beginBulk()`, `isBudgetTight()`, and a `CHECK`
+constraint hand-synced to a PHP constant. Rationale: that precision existed to guarantee
+≤495 requests per rolling minute — yet on saturation the limiter logged and **proceeded unmetered
+anyway**, so the exact bound was unenforceable by design and the arithmetic bought nothing.
+
+## Gotchas / known issues
+
+- `_Query::setQuery()` **executes non-SELECT queries immediately** — `new _Query($writeSql, $alias)`
+  runs the write eagerly, with no `execute()` call; `__destruct()` calls `execute()` as a backstop.
+  This is why any async flag must be set *before* `setQuery()`
+  (see [Asynchronous Query Execution](./async-query-execution.md)).
+- `_Query::fetchRow()` returns an **object**, not an array — `$row->requests`, never
+  `$row['requests']`. The affected-rows accessor is **`getAffectedRows()`** (there is no
+  `affectedRows()`). `fetchOne()` returns the first column of the first row, or `false` — the
+  cleanest single-value read.
+- **Security (location only):** `_underscore/Component/Api/Clickup/Clickup.php` declares a
+  hardcoded shared ClickUp workspace access token as a class constant next to `ENDPOINT`. Every
+  consumer authenticates as the same workspace identity, so per-consumer attribution and isolation
+  are impossible at the provider end — which also means a budget can never be attributed per caller
+  upstream. Needs rotation, a move to config/SSM, and a `/cso` pass. Already noted in
+  [ClickUp Design Sprint Automation](../../worker2/features/clickup-design-sprint-automation.md).
+
+## Ticket mapping (so the next reader does not conflate these)
+
+- **TRUE-80662 "ClickUp - Fix Project Hub Routing"** (Sprint 83, back-end review; reviewer Jeff
+  Cardinal) — contains nothing about rate limiting. The limiter arrived *inside* its PRs (#137,
+  #449). The outbound budget work has **no ticket of its own yet**.
+- **TRUE-80659** "formal rate limiting feature for the API … return a 429 with a Retry-Again
+  header" (filed by Jeff Cardinal; Backlog/On Bench, unassigned, unbuilt) — that is the **inbound**
+  429 + `Retry-After` feature, a **different** concern from this outbound budget.
+- **TRUE-4193** — an older per-IP **inbound** variant.
+
+## Change history
+- 2026-09-02 — Created. Designed `_Component_Api_RateBudget` + `Core.ApiRateBudget` (staged, not
+  merged) as a provider-agnostic replacement for the rejected 639-line ClickUp-only limiter;
+  recorded the false 500/min premise, the ~24 unmetered ClickUp call sites, the blind enforcement
+  test, the dedicated-alias requirement for any cross-process counter, and the no-sleep-wait
+  enforcement tiering. (ajean)
+
+## Related docs
+- [_underscore Framework Architecture](../architecture.md)
+- [Creating Worker Actions](../../worker2/features/creating-worker-actions.md) — the 300s cap
+- [ClickUp Project & Opportunity Multi-List Routing](../../worker2/features/clickup-project-routing.md) — PR #137's actual subject
+- [Asynchronous Query Execution](./async-query-execution.md)
+- [_Model::save() vs raw _Query](./model-save-vs-query-atomic-update.md)

package/knowledge/2.0/apps/worker2/features/clickup-project-routing.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-31
+updated: 2026-09-02
 owners: [jcardinal, ajean]
 files:
   - worker2/Worker/Clickup/Project.php
@@ -17,6 +17,7 @@ related:
   - ./netsuite-opportunity-sync.md
   - ./clickup-connectivity-watchdog.md
   - ./clickup-subtask-activity.md
+  - ../../_underscore/features/outbound-api-rate-budget.md
 ---
 
 ## Summary
@@ -166,6 +167,14 @@ None — uniform across all clients (shared internal automation for Agilant's Cl
   needs **`taskCreated` on LS + A&M** and a **workspace-wide `taskMoved`**, registered via
   `RegisterHubGuardWebhooks()` (which skips any already covered to avoid double-delivery).
 
+- **PR #137 also carries a ClickUp rate limiter that is NOT part of this feature — and it was
+  rejected in review.** The open branch `fix/clickup-project-hub-routing` bundles a 533-line
+  `Worker/Clickup/Api.php` limiter (plus dbchanges2 #449) whose premise, coverage, and self-test are
+  all wrong: api2 and tools do **not** call ClickUp (verified by grep), ~24 of ~116 call sites stay
+  unmetered, and its `TestNoDirectApiCalls()` guard cannot see the 18 hand-built `api.clickup.com`
+  sites. Treat rate limiting as a separate subject with its own design —
+  [Outbound third-party API rate budget](../../_underscore/features/outbound-api-rate-budget.md).
+
 ## Diagnostics
 
 Read-only / setup actions, invoked via `curl -X POST https://worker.togahub.com/` with
@@ -176,6 +185,8 @@ Read-only / setup actions, invoked via `curl -X POST https://worker.togahub.com/
 `RESULT: PASS`), `RegisterOpportunityWebhook` and `RegisterHubGuardWebhooks` (one-time setup).
 
 ## Change history
+- 2026-09-02 — Noted that PR #137 bundles a rejected ClickUp rate limiter unrelated to routing, and
+  split that subject out to its own doc. No routing behavior changed. (ajean)
 - 2026-08-31 — Recorded the triage consequence of the hybrid split (read-only investigation, no code
   changed): worker-created opportunity tasks were observed **homed in `901111987451`
   "Quote/Proposal/SOW/MSA"** rather than the `901111987449` list they were POSTed to, with the mover

package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md

@@ -17,6 +17,7 @@ related:
   - ../architecture.md
   - ./netsuite-opportunity-client-labels.md
   - ../../api2/features/api-payload-interceptors.md
+  - ../../_underscore/features/outbound-api-rate-budget.md
 ---
 
 ## Summary
@@ -232,6 +233,13 @@ production (2026-08-28, job `1075223`, a full `OpportunityStakeholderBackfill` d
 So an over-budget action loses its output **twice**: once to a false failure row, once to a dropped
 connection. Design for 300s.
 
+**Corollary — never sleep-wait for an external rate budget inside an action.** Pausing to wait for
+a third-party API slot burns the 300s cap, gets the job stamped failed by the watchdog, and there is
+no retry and no DLQ to recover it. Instead: **bulk/sweep** actions yield at the threshold and
+re-enqueue the remainder (they are already chunked and resumable, so pausing is free), while
+**event-driven/escalation** actions proceed over budget, counted and alarmed. See
+[Outbound third-party API rate budget](../../_underscore/features/outbound-api-rate-budget.md).
+
 ### Self-chunking: the pattern for work that cannot finish in one job
 
 An action that may exceed the budget works for a **sub-budget slice**, then **re-enqueues itself**
@@ -456,6 +464,9 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
   commit-before-SQS transaction pattern that the worker relies on.
 
 ## Change history
+- 2026-09-02 — Added the corollary that an action must **never sleep-wait for a third-party API
+  rate budget** (it burns the 300s cap and the failure is unrecoverable — no retry, no DLQ); bulk
+  work yields and re-enqueues, event-driven work proceeds over budget counted and alarmed. (ajean)
 - 2026-09-02 — Added two authoring rules learned on the End Customer label work (worker2 PR #158/#159):
   **gate any code that names a column from a hand-applied dbchanges2 migration** on an
   `information_schema.COLUMNS` check memoised in a `static` (an ungated write fails the whole query

package/knowledge/2.0/standards/backend-php.md

@@ -5,8 +5,8 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-09-01
-owners: [jcardinal, mhammontree, dfranks, bala]
+updated: 2026-09-02
+owners: [jcardinal, mhammontree, dfranks, bala, ajean]
 files: []
 related:
   - ../apps/_underscore/architecture.md
@@ -521,6 +521,33 @@ unapproved for 6 days because the approval was stamped with a dead user id. Prod
 See: clients/compass-usa/features/approval-decision-flow.md and
 clients/compass-usa/features/stranded-approval-reassignment.md
 
+### A cross-process counter or shared rollup MUST use its own DB alias, committed immediately
+
+`Route.php` opens a transaction on the **default** connection and holds it open until the request
+or job ends. So anything you write on the default alias for other processes to read is broken twice
+over:
+
+1. **Nobody else can see it** until your job ends — a fleet-wide total is unreadable by design.
+2. **You hold a write lock on one hot row** for up to the full 300s job budget, serialising every
+   other process that touches that counter.
+
+So any shared counter, rate budget, sequence, or rollup must:
+
+- register a **second connection under its own alias**, and
+- call `_Database::transactionCommit($alias)` **immediately after the write** — not at end of job.
+
+Removing the extra alias looks like a simplification. It is not: it silently breaks correctness and
+adds a fleet-wide lock.
+
+**Read the value back by primary key after the upsert.** Do not use the one-statement
+`LAST_INSERT_ID(expr)` upsert-and-read idiom on a table with an `AUTO_INCREMENT id` (which our
+table standard mandates): `LAST_INSERT_ID()` returns the new row id on the INSERT path but the
+assigned expression on the UPDATE path, so the result is path- and version-dependent. One extra
+indexed lookup is obviously correct. Under concurrency the read-after-write skew is upward-only,
+so the number is conservative.
+
+See [Outbound third-party API rate budget](../apps/_underscore/features/outbound-api-rate-budget.md).
+
 ## PHP
 
 ### PHP Tags
@@ -997,6 +1024,15 @@ See: 2.0/apps/worker2/features/cross-account-aws-access.md
 
 ## Change history
 
+- 2026-09-02 — Database/SQL: any **cross-process counter or shared rollup needs its own DB alias,
+  committed immediately**. `Route.php` holds a transaction open on the default connection until the
+  request/job ends, so a counter written there is invisible to every other process (a fleet-wide
+  total is unreadable) and holds a write lock on one hot row for up to the 300s job budget,
+  serialising the fleet. Register a second connection under its own alias and
+  `_Database::transactionCommit($alias)` right after the write. Also: read the value back by
+  primary key instead of the one-statement `LAST_INSERT_ID(expr)` upsert-and-read idiom — with the
+  mandated `AUTO_INCREMENT id` it returns the new row id on INSERT but the assigned expression on
+  UPDATE, so it is path- and version-dependent. (ajean)
 - 2026-09-01 — TRUE-81576. Three additions: (1) Security — never ship a hardcoded-credential
   fallback; the FedEx `initialize()` "production fallback" holds a live `client_id`/`client_secret`
   byte-identical to `api2/Config/production.ini`, so it must throw on missing `[fedex]` config, the

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 75 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 76 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 58 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 15 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.740",
+  "version": "1.0.741",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",