toga-ai 1.0.284 → 1.0.286

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -6,6 +6,7 @@
 | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Page.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql |
 | [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php |
 | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
+| [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | `_Query` can run a **write** query asynchronously so a long/slow write does not hold a request-scoped DB connection open long enough to hit **"MySQL server has | _underscore/Query.php, worker2/Worker/Infrastructure/Database.php, worker2/Worker/Team/Transcripts.php |
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/ShipmentRequest/ShipmentRequest.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
 | [Error Reporting — Issue/Event Aggregation (exceptionHandler)](features/error-reporting-issue-event.md) | `_underscore`'s global exception handler persists every uncaught exception into a two-table **Issue / Event** model in the **shared Core Logs DB** (`_underscore | _underscore/Error.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql |

package/knowledge/2.0/apps/_underscore/architecture.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: architecture
 status: active
-updated: 2026-07-02
+updated: 2026-07-07
 owners: ["jcardinal", "rgirish", "mhammontree"]
 files:
   - _underscore/_underscore.php
@@ -119,6 +119,16 @@ multi-tenant client-DB lookup from Core. `Query.php` wraps MySQLi with read/writ
 auto-routing and result helpers: `fetchRows()`, `fetchRow()`, `fetchOne()`,
 `fetchRowsAssocArray()`, `fetchKeyValuePairs()`, `fetchValues()`.
 
+`_Query` also supports a **writes-only async mode** (`isAsync` — 3rd constructor arg /
+`setIsAsync()`): a slow WRITE is dispatched to the Worker tier via `_Worker::runTask()`
+instead of running on the request connection, so a long write cannot hold a request-scoped
+connection open long enough to hit **"MySQL server has gone away."** See
+[features/async-query-execution.md](./features/async-query-execution.md). This rides on the
+**SQS-first** `_Worker::runTask()` enqueue path (below): `runTask()` no longer INSERTs the
+`WorkerJobs` row on the caller's connection — it sends `{uuid, action, parameters}` to the
+worker SQS queue and the worker tier creates/tracks the row on its own fresh connection at
+pickup (see the worker2 architecture doc's SQS-first decision).
+
 ### Physical infrastructure
 
 Each **environment** is one or more MySQL clusters. Every cluster has a single
@@ -338,3 +348,4 @@ multi-file UI components (`.php`/`.html`/`.css`/`.js`) invoked as `<_ComponentNa
 - 2026-06-25 — Added the Surface platform UI presentation/configuration layer (DB-driven UI config replacing `Page::meta()`, CTO-reviewed AGREE-WITH-ADJUSTMENTS) (jcardinal)
 - 2026-06-29 — Surface made the enforced (un-flagged) presentation path; reserved id blocks renumbered to Records 333–341 / RecordFields 2246–2433 (known seed-vs-provisioned drift). (jcardinal)
 - 2026-07-02 — Fixed framework-wide `FIELD_STORAGE` write-drop and folder-read bugs in `Model.php`; noted the remaining unconditional-refetch dirty-read follow-up. (mhammontree)
+- 2026-07-07 — Documented `_Query` writes-only async mode (`isAsync` → Worker `Infrastructure/Database/Query`) in the database-architecture section, linked `features/async-query-execution.md`, and noted the SQS-first `_Worker::runTask()` enqueue departure (caller does no MySQL; worker tier creates the row on pickup). (jcardinal)

package/knowledge/2.0/apps/_underscore/features/async-query-execution.md

@@ -0,0 +1,89 @@
+---
+title: Asynchronous Query Execution (writes-only, via Worker)
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-07-07
+owners: ["jcardinal"]
+files:
+  - _underscore/Query.php
+  - worker2/Worker/Infrastructure/Database.php
+  - worker2/Worker/Team/Transcripts.php
+related:
+  - ../architecture.md
+  - ./per-client-database-connections.md
+---
+
+## Summary
+
+`_Query` can run a **write** query asynchronously so a long/slow write does not hold a
+request-scoped DB connection open long enough to hit **"MySQL server has gone away"**.
+Instead of executing on the local connection, the query is dispatched as a Worker task
+(`Infrastructure/Database/Query`) that opens its own connection on the target DB, runs the
+write, and commits. It is **writes-only by design** — an async SELECT captures no result set
+and is a harmless no-op.
+
+## Key files / entry points
+
+- **`_underscore/Query.php`** — `_Query` gained an `isAsync` flag: constructor 3rd param
+  `bool $isAsync = false`, a `setIsAsync()` setter, and an `_isAsync` property. When async,
+  `execute()` resolves the target DB connection config and dispatches a Worker task instead of
+  running locally.
+- **`worker2/Worker/Infrastructure/Database.php`** — `_Worker_Infrastructure_Database::Query()`
+  is the worker-side action that runs the write on the target DB and commits.
+- **`worker2/Worker/Team/Transcripts.php`** — live example caller in `SyncKnowledgeBases()`
+  (lines ~911 / ~925).
+
+## How it works
+
+1. **Opt in.** Construct with the third arg true:
+   `new _Query($sql, _underscore::DB_TEAM, true);` (or call `setIsAsync(true)` before the
+   query is set).
+2. **Dispatch (in `execute()`).** When `_isAsync` is true, `execute()` does **not** run the
+   query locally. It resolves the target DB connection config — from `_Database::$_registers`
+   or the `_Config` `DATABASE` group (direct hostname, or `writehost`/`readhost` + `backuphost`
+   params) — and calls
+   `_Worker::runTask(action: 'Infrastructure/Database/Query', parameters: {hostname, database,
+   username, password, query})`.
+3. **Worker side.** `_Worker_Infrastructure_Database::Query()` calls `_Database::register(...)`
+   under the alias `'Query'`, runs `new _Query($query, 'Query')` to execute the write on the
+   worker's own connection, then commits with `_Database::transactionCommit('Query')`.
+4. **Writes only.** Async SELECT captures no result set (fire-and-forget) and is treated as a
+   harmless no-op — do not use async for reads.
+
+## Correctness gotchas (all fixed this session — do not regress)
+
+- **Constructor order (critical).** `setIsAsync()` must run **before** `setQuery()`, because
+  `setQuery()` executes non-SELECT queries immediately. With the original order (setQuery first)
+  every async WRITE ran synchronously and the async path was never taken. The constructor was
+  reordered to set the async flag first.
+- **Worker-side commit (critical).** `_Database::register()` with an alias auto-starts a lazy
+  transaction; the write flips autocommit off and is rolled back when the worker connection
+  closes unless explicitly committed. The worker action must call
+  `_Database::transactionCommit('Query')` after the write. This is the known _underscore
+  [lazy-transaction gotcha](../architecture.md) — the same trap that silently discarded Rate
+  SAML user INSERTs.
+- **Double-dispatch.** The async branch leaves `_results` null, and `execute()` is guarded by
+  `is_null(_results)`, so `__destruct()`/getters re-fire the job. Fixed by setting a sentinel
+  (`_results = true`) at the end of the async branch.
+- **Dangling local transaction.** The original async check ran **after** the
+  write-connection/transaction setup, so a dispatching async write opened a local writer
+  connection and started a local transaction it never committed. Fixed by moving the async
+  dispatch to the **top** of `execute()`, before any read/write connection or transaction setup.
+
+## Known open items
+
+- **SECURITY (open, developer aware).** DB username/password are currently sent in **plaintext**
+  through the Worker queue payload (SQS + `WorkerJobs` table). Being handled separately: the
+  worker should resolve credentials from config by database name rather than receiving them in
+  the payload. Do not build on the plaintext-credential path.
+
+## Change history
+
+- 2026-07-07 — Built writes-only async query execution (`_Query` `isAsync` → Worker
+  `Infrastructure/Database/Query` action); fixed four correctness bugs found in review
+  (constructor order, worker-side commit, double-dispatch sentinel, dangling local
+  transaction). Credential-in-payload security item left open. (jcardinal)

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-06-08
+updated: 2026-07-07
 owners: [jcardinal]
 files:
   - worker2/Controller/Index.php
@@ -15,6 +15,7 @@ files:
   - _underscore/Worker.php
 related:
   - ./features/creating-worker-actions.md
+  - ../_underscore/features/async-query-execution.md
 ---
 
 ## Summary
@@ -92,8 +93,29 @@ New path (`{workerJobId}` present):
 5. Call `initialize()` if present, then the action with spread parameters.
 6. **Check-out:** `UPDATE dtCompleted, executionTime, isSuccess, failureReason`. Return 200.
 
-Legacy path (`action` present, no `workerJobId`): used by `_Worker::runTask()` debug mode
-only; runs directly with no `WorkerJobs` tracking.
+Direct-payload path (`action` present, no `workerJobId`): formerly the debug-only "legacy"
+path (untracked, could return HTTP 500). As of 2026-07-07 it is a **first-class tracked
+path** — it is how the SQS-first `_Worker::runTask()` (below) delivers production jobs. The
+message carries `{uuid, action, parameters}`; the worker creates and tracks the `WorkerJobs`
+row on its own fresh connection and **always returns HTTP 200** (no DLQ + 3600s visibility
+timeout means any 500 = poison-message storm):
+
+1. **Dedupe guard.** SELECT `WorkerJobs` by `uuid`. Skip (200) **only if `isSuccess IS NOT
+   NULL`** (already completed). A row that exists but is incomplete (prior crash / in-flight)
+   is **re-run**, not dropped — mirrors the `workerJobId` guard. (SQS is at-least-once.)
+2. **New job → INSERT** `(uuid, jobType='ACTION', action, parameters, dtQueued=NOW(),
+   dtStarted=NOW(), instanceId)`. A UNIQUE index on `WorkerJobs.uuid` (added in `dbchanges2`)
+   backstops the SELECT→INSERT TOCTOU race.
+3. **INSERT-failure disambiguation** is by **re-resolving the uuid**, not by parsing the
+   error code: if the row now exists it was a benign concurrent duplicate (skip 200);
+   otherwise the write genuinely failed — log CRITICAL + `\Sentry\captureException`, still
+   return 200.
+4. `dtQueued=NOW()` at insert (not NULL) so JobScheduler's `dtQueued IS NULL` recovery does
+   not re-queue these rows; the watchdog still covers them via `dtStarted`.
+
+> **Known follow-up (not done):** the ~100-line log-DB-registration + `initialize()` blocks
+> are duplicated between the `workerJobId` path and the direct-payload path — candidate for
+> extraction to shared private helpers.
 
 ## Action → class/method routing
 
@@ -103,9 +125,17 @@ file `Worker/Team/Github.php`.
 
 ## Programmatic invocation & manual retry
 
-- **`_Worker::runTask(action, parameters)`** (`_underscore/Worker.php`) — INSERT
-  `WorkerJobs` → **`transactionCommit(DB_CORE)` before SQS** → send `{workerJobId}`.
-  Debug mode posts synchronously with no tracking.
+- **`_Worker::runTask(action, parameters)`** (`_underscore/Worker.php`) — **SQS-first as of
+  2026-07-07:** the production branch does **no MySQL** (no INSERT, no commit). It generates a
+  client-side `uuid` and sends `{uuid, action, parameters}` to the worker SQS queue; the worker
+  tier creates+tracks the `WorkerJobs` row on pickup (see direct-payload path above). Payload is
+  capped at `_Worker::SQS_MAX_MESSAGE_BYTES = 262144` — `runTask()` **throws** if the JSON
+  exceeds the SQS 256KB limit (loud failure, not a silent drop); oversized params must be
+  offloaded to S3 by the caller. Debug branch unchanged (synchronous POST). **Why:** long-running
+  scripts hit "MySQL server has gone away" on the caller's stale Core connection during the old
+  INSERT; SQS sends do not depend on the DB connection, so row creation moved off the caller
+  entirely. This is the enqueue path used by `_Query` writes-only async mode. See the SQS-first
+  decision + trade-off under *Key design decisions*.
 - **`_Worker_Infrastructure_Worker::Retry($workerJobId)`** — resets the job fields,
   commits before SQS, sends `{workerJobId}` immediately. Sets `dtQueued=NOW()` (not NULL)
   so JobScheduler doesn't double-pick it.
@@ -118,17 +148,56 @@ file `Worker/Team/Github.php`.
 ## Critical transaction pattern
 
 **Any INSERT/UPDATE that must be visible to another connection before an SQS message is
-delivered must be immediately committed.** Three sites: `Retry()`, `_Worker::runTask()`,
-and the `Controller/Index.php` check-in. Reason: the caller's MySQL transaction is still
+delivered must be immediately committed.** Sites: `Retry()` and the `Controller/Index.php`
+check-in. (`_Worker::runTask()` no longer applies as of 2026-07-07 — it does no MySQL; the
+worker tier creates the row itself on pickup.) Reason: the caller's MySQL transaction is still
 open when SQS delivers; the worker reads in a separate connection and, if uncommitted, its
 SELECT returns nothing and the guard fires ("already processed or not found — skipping").
 
 ## Key design decisions
 
-MySQL-first (SQS on success) prevents lost jobs · always HTTP 200 (WorkerJobs tracks
-failures, SQS retry unwanted) · 3600s visibility timeout for hour-long jobs · no DLQ
-(developer-controlled `Retry()` preferred) · all fallback payloads to S3 (no size split) ·
-SQS failure → reset `dtQueued=NULL` (rescheduled next minute) · `maxExecutionTime` on
-CronJobs, env var for ACTION jobs · `/Webhook` appended enforces method-name convention ·
-API Gateway in front of Lambda (SCP blocks Function URLs) · `workerJobId` lookup keeps SQS
-messages tiny · commit before SQS avoids the delivery-before-commit race.
+MySQL-first for the **Lambda/webhook/cron ingestion paths** (SQS on success) prevents lost
+jobs · always HTTP 200 (WorkerJobs tracks failures, SQS retry unwanted) · 3600s visibility
+timeout for hour-long jobs · no DLQ (developer-controlled `Retry()` preferred) · all fallback
+payloads to S3 (no size split) · SQS failure → reset `dtQueued=NULL` (rescheduled next
+minute) · `maxExecutionTime` on CronJobs, env var for ACTION jobs · `/Webhook` appended
+enforces method-name convention · API Gateway in front of Lambda (SCP blocks Function URLs) ·
+`workerJobId` lookup keeps SQS messages tiny · commit before SQS avoids the
+delivery-before-commit race.
+
+### `_Worker::runTask()` is SQS-first, NOT MySQL-first (2026-07-07 — intentional departure)
+
+For **programmatic** enqueue via `_Worker::runTask()`, the MySQL-first rule was deliberately
+reversed. Long-running PHP scripts were hitting "MySQL server has gone away" on the caller's
+`WorkerJobs` INSERT because the caller's Core connection had gone **stale** (not because the
+server was dying). Approach chosen (**A — SQS-first**): the caller does no MySQL; it sends
+`{uuid, action, parameters}` to SQS and the worker tier creates+tracks the row on its own
+fresh connection at pickup.
+
+**Rejected alternative (CTO-recommended):** Approach C/D — reconnect-on-stale inside
+`_Database`, keeping the durable id-based (MySQL-first) pipeline. The CTO review advised
+**against** Approach A because the failure is a stale PHP connection rather than a dying
+server, and because promoting the debug-only untracked path risked poison-message storms and
+duplicate execution. The developer chose Approach A anyway, **accepting the durability
+trade-off**: a bare SQS-send failure loses the job, and jobs are not visible in `WorkerJobs`
+until worker pickup. The CTO review's known-danger mitigations were then implemented — uuid
+dedupe that re-runs incomplete rows, UNIQUE `uuid` index for the TOCTOU race, uuid
+re-resolution to disambiguate INSERT failures (CRITICAL log + Sentry on genuine failure),
+always-200, and a 256KB payload cap that throws. Record kept so a future reader understands
+the MySQL-first design was departed from **on purpose** here.
+
+**Gotchas from this change:**
+- `dtQueued` on these rows now reflects worker pickup/row-creation time, **not** true SQS
+  enqueue time — semantic drift for any queue-latency metric.
+- Debug/legacy messages without a `uuid` get a random one, so dedupe does not apply to them —
+  an HTTP-layer retry of a debug call could double-run.
+
+## Change history
+
+- 2026-06-08 — Initial worker2 architecture doc. (jcardinal)
+- 2026-07-07 — `_Worker::runTask()` moved to SQS-first (caller does no MySQL; worker tier
+  creates/tracks the row on pickup); the former debug-only direct-payload path in
+  `Controller/Index.php` promoted to a first-class tracked path with uuid dedupe, UNIQUE-index
+  TOCTOU backstop, uuid re-resolution on INSERT failure, and a 256KB payload cap. Documented as
+  an intentional departure from MySQL-first (CTO recommended against; trade-off accepted).
+  Related: `_underscore` `_Query` writes-only async mode. (jcardinal)

package/knowledge/2.0/apps/worker2/features/clickup-work-type-automation.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-23
+updated: 2026-07-08
 owners: ["jcardinal"]
 files:
   - worker2/Worker/Clickup.php
@@ -69,8 +69,26 @@ None — internal team/sprint tooling, uniform across clients.
 - **No PHPUnit harness** in worker2. The regression test
   `Tests/Worker/ClickupWorkTypeTest.php` is a plain-PHP script (reflection on
   `isStatusComplete`) run with `php Tests/Worker/ClickupWorkTypeTest.php`.
+- **ClickUp dropdown custom-field `value` is the option ORDERINDEX, not its name.** The
+  webhook payload for a dropdown/labels custom field delivers the selected option's integer
+  `orderindex`, *not* the option's display name. Any name-keyed lookup (e.g.
+  `$lookupPreviousSprintWorkTypeByName`) will silently never match if fed the raw `value`.
+  Resolve the name first via the field's own `type_config->options` — build a map keyed by
+  `orderindex` (`$workTypesByIndex[$opt->orderindex] = $opt`) and read `->name` from it. This
+  is the proven pattern in `_Worker_Team_Sprint::SprintLock` (`Worker/Team/Sprint.php`, the
+  Work Type resolve ~L3894 and the POST ~L3971). Guard the lookup with
+  `isset($workTypesByIndex[$value])` so a stale/deleted option orderindex doesn't warn.
 
 ## Change history
+- 2026-07-08 — Fixed **Previous Sprint Work Type** never being set on tasks created after a
+  sprint launched (`taskCreated` case, `Worker/Clickup.php` ~L715-751). Root cause: the handler
+  assigned `$workType = $customField->value` (the ClickUp option orderindex, an int) then looked
+  it up in a name-keyed map, so the `isset()` guard never matched and the POST silently never
+  fired. Now resolves orderindex→name via the WORKTYPE field's `type_config->options`
+  (`$workTypesByIndex`), matching the `SprintLock` convention, with an added
+  `isset($workTypesByIndex[$value])` guard. Also added: empty Work Type now defaults to
+  `Unplanned` before the Previous Sprint Work Type lookup so newly-created tasks still get a
+  value. `php -l` clean; php-reviewer approved. (jcardinal)
 - 2026-06-23 — Fixed dependents staying Conditional after a blocker completed: completion check
   now treats `closed` as terminal alongside `done`, and `taskStatusUpdated` re-evaluates
   dependents (gated to terminal statuses). Added `COMPLETE_STATUS_TYPES`, `isStatusComplete()`,

package/knowledge/INDEX.md

@@ -17,7 +17,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 24 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 25 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 27 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 10 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.284",
+  "version": "1.0.286",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",