toga-ai 1.0.450 → 1.0.451

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -6,7 +6,7 @@
 | [_underscore Framework Architecture](architecture.md) | `_underscore` is the shared PHP backend framework for **all 2.0 applications**. | _underscore/_underscore.php, _underscore/Loader.php, _underscore/Framework.php, _underscore/Model.php, _underscore/Database.php, _underscore/Query.php, _underscore/Route.php, _underscore/Component.php |
 | [ACL Permission Chain (Record & Field Authorization)](features/acl-permission-chain.md) | Authorization in the 2.0 API is **metadata-driven**: whether a role may Create/Read/Update/Delete a record is decided by rows across **four linked tables**, not | api2/Component/Api/V2/V2.php, _underscore/Model/Core/Page.php, _underscore/Model/Client/TrackingNumber.php, dbchanges2/Client/2026-06-03- BLANK_CLIENT_DATABASE.sql, dbchanges2/Client/2026-06-23b - ItemTranslationsAcl.sql, dbchanges2/Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql |
 | [Address Validation (carrier waterfall + validateAddress scripted endpoint)](features/address-validation.md) | `_Model_Client_Address::validateAddress` verifies a US address against a **carrier waterfall (USPS → FedEx → UPS)** and returns a single canonical, carrier-norm | _underscore/Model/Client/Address.php |
-| [_ApiRequest ENCODE__JSON now sends Content-Type: application/json](features/apirequest-json-content-type.md) | `_ApiRequest::execute()`'s `ENCODE__JSON` branch json-encoded the request body but never set a `Content-Type` header. | _underscore/ApiRequest.php |
+| [_ApiRequest — JSON encode/decode & api-logging behavior](features/apirequest-json-content-type.md) | `_ApiRequest` is the 2.0 outbound HTTP client. | _underscore/ApiRequest.php |
 | [Assortment Name Translation (AssortmentTranslations sidecar)](features/assortment-name-translation.md) | Serves Assortment (product-grouping) **names** in multiple languages by adding a per-language **sidecar** table `AssortmentTranslations`, reusing the platform's | _underscore/Model/Client/AssortmentTranslation.php, dbchanges2/Client/2026-06-26a - AssortmentTranslations.sql, dbchanges2/Core/2026-06-26a - AssortmentTranslationsRecord.sql, dbchanges2/Client/2026-06-26b - AssortmentTranslationsAcl.sql |
 | [Asynchronous Query Execution (writes-only, via Worker)](features/async-query-execution.md) | `_Query` can run a **write** query asynchronously so a long/slow write does not hold a request-scoped DB connection open long enough to hit **"MySQL server has | _underscore/Query.php, worker2/Worker/Infrastructure/Database.php, worker2/Worker/Team/Transcripts.php |
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/ShipmentRequest/ShipmentRequest.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |

package/knowledge/2.0/apps/_underscore/features/apirequest-json-content-type.md

@@ -1,21 +1,27 @@
 ---
-title: _ApiRequest ENCODE__JSON now sends Content-Type: application/json
+title: "_ApiRequest — JSON encode/decode & api-logging behavior"
 framework: "2.0"
 repo: _underscore
 project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-14
+updated: 2026-07-28
 owners: ["jcardinal"]
 files:
   - _underscore/ApiRequest.php
 related:
   - ../../worker2/features/oneuptime-worker2-monitoring.md
+  - ../../worker2/features/creating-worker-actions.md
+  - ../../worker2/features/talos-transcript-ingestion.md
 ---
 
 ## Summary
-`_ApiRequest::execute()`'s `ENCODE__JSON` branch json-encoded the request body but never set
+`_ApiRequest` is the 2.0 outbound HTTP client. This doc collects its non-obvious
+encode/decode/retry/logging behavior that callers keep getting bitten by; the original subject
+was the Content-Type fix below.
+
+**Original fix (2026-07-14):** `_ApiRequest::execute()`'s `ENCODE__JSON` branch json-encoded the request body but never set
 a `Content-Type` header. cURL therefore defaulted to `application/x-www-form-urlencoded`, and
 receivers parsed the entire JSON string as a single form-field NAME — observed at OneUptime as
 `{"<json>":""}`. Every 2.0 `ENCODE__JSON` POST/PUT/PATCH caller was silently shipping
@@ -30,10 +36,40 @@ case-insensitive scan of already-set headers, so a caller-supplied `Content-Type
 - This changes the wire format of ALL existing `ENCODE__JSON` callers (they were previously
   sending form-urlencoded). This is a correction, but smoke-test heavy JSON callers post-deploy:
   ClickUp, NetSuite, Vapi.
+- **`execute()` decodes a JSON response with `json_decode($payload)` — NO assoc flag — so you
+  get `stdClass`, not an array.** Code that indexes the response as `$res['result']['x']` breaks.
+  If a caller needs arrays, don't rely on the built-in decode: leave the response handling local
+  (`json_decode($response, true)`).
+- **`ENCODE__JSON` re-encodes the payload without `JSON_UNESCAPED_SLASHES|JSON_UNESCAPED_UNICODE`.**
+  For large non-ASCII bodies (e.g. a full meeting transcript) this inflates every such char to
+  `\uXXXX`. To preserve the exact wire bytes, `json_encode` with the flags you want yourself and
+  pass the **ready JSON string with payloadEncoding left null**.
+- **`setAutoRetry()` is flat-delay and retries EVERY non-2xx.** If you need exponential backoff
+  or fail-fast on 4xx (other than 429), keep your own retry loop and construct one `_ApiRequest`
+  per attempt — that also gives you one api-log row per attempt.
+- **Transport failures surface as a thrown Exception** from `execute()` (the equivalent of
+  `curl_error()`), not a return value. Catch it if your method has an error-return contract.
+- **Api logging depends on `DB_CLIENT_LOGS` being registered.** The logging branch writes via
+  `_Model_Client_Logs_Api`, whose `DATABASE` const is `_underscore::DB_CLIENT_LOGS`. In a context
+  that hasn't registered it (most workers, CLI harnesses) logging is a **silent no-op** — or
+  throws `Unknown database 'ClientLogs'`. Either register the logs DB under that alias or pass
+  `setLogging(false)` deliberately. See
+  [Creating Worker Actions](../../worker2/features/creating-worker-actions.md#gotchas).
+- **OPEN / not fixed: logging happens in two halves around the HTTP call.** `execute()` inserts
+  the request row and **commits before** the call, then `save()`s the response fields **after**.
+  For long calls (e.g. a 600s AI timeout) that second save runs on a connection idle for
+  minutes — the classic stale-connection failure mode — leaving api-log rows with null
+  `responseCode`/`responsePayload`. The correct fix (reconnect/re-register before the post-call
+  save) lives **inside `execute()`** and would affect every framework caller, so it is deferred
+  pending architecture review. Do **not** work around it per-caller.
 
 ## Change history
+- 2026-07-28 — Broadened from the Content-Type fix to the general `_ApiRequest` contract:
+  documented the non-assoc `json_decode` response (returns `stdClass`), the `ENCODE__JSON`
+  re-encode losing `JSON_UNESCAPED_*`, `setAutoRetry()`'s flat-delay/retry-all behavior,
+  exception-on-transport-failure, the `DB_CLIENT_LOGS` dependency for api logging, and the
+  **open** two-phase logging / stale-connection issue on long calls. Surfaced migrating
+  `worker2 Worker/Team/Transcripts.php::callTalosEndpoint()` off raw curl. (jcardinal)
 - 2026-07-14 — Fixed `ENCODE__JSON` to send `Content-Type: application/json` (guarded so a
   caller-set Content-Type wins). Root-caused via OneUptime receiving `{"<json>":""}`. Affects
   all 2.0 ENCODE__JSON callers. (jcardinal)
-</content>
-</invoke>

package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-21
+updated: 2026-07-28
 owners: [jcardinal, dfranks, mhammontree]
 files:
   - worker2/Worker/
@@ -189,10 +189,29 @@ be reattempted.
   `$environment = (substr(_Environment::$name, 0, 4) === 'dev-') ? 'dev' : _Environment::$name`.
   Without it, every `_Model_Client_*` load fails. Proven pattern: `_Worker_Startech` (and
   `_Worker_Notification_EmailTemplate`). Do this in `initialize()` or at the top of the method.
+- **`_ApiRequest` logging is a silent no-op unless the Logs DB is registered as
+  `DB_CLIENT_LOGS`.** `_ApiRequest`'s logging branch writes through `_Model_Client_Logs_Api`,
+  whose `DATABASE` const is `_underscore::DB_CLIENT_LOGS`. A worker whose `initialize()` only
+  registers, say, `DB_TEAM` will produce **zero** api-log rows (or, in a CLI/test harness,
+  `Unknown database 'ClientLogs'`). So for any worker that makes outbound `_ApiRequest` calls,
+  pick one deliberately:
+  - **Want the calls logged** → register the logs DB in `initialize()`, e.g.
+    `_Database::registerDatabase(_Config::databaseLogs('Logs_True', …), _underscore::DB_CLIENT_LOGS)`
+    (proven pattern: `Worker/Team/Sprint.php::initialize()`, `Worker/Team/Transcripts.php::initialize()`).
+  - **Don't want them logged** → pass `setLogging(false)` explicitly, with a comment saying why
+    (proven: `Worker/Clickup/Fluffer.php`, `Worker/Vapi.php`, `Worker/Ai/Bdr/Vapi.php`).
+
+  Swapping raw curl for `_ApiRequest` "to get logging" without doing the first of those is the
+  common trap — the code looks right and logs nothing.
 - See [architecture.md](../architecture.md) for the always-HTTP-200 rule and the
   commit-before-SQS transaction pattern that the worker relies on.
 
 ## Change history
+- 2026-07-28 — Added the **`_ApiRequest` logging / `DB_CLIENT_LOGS` gotcha**: api-log rows are
+  written via `_Model_Client_Logs_Api` (`DATABASE = DB_CLIENT_LOGS`), so a worker must either
+  register the Logs DB under that alias in `initialize()` or pass `setLogging(false)`
+  deliberately. Surfaced migrating the Talos AI caller in `Worker/Team/Transcripts.php` off raw
+  curl. (jcardinal)
 - 2026-07-21 — Documented that adding a recurring job is **pure data** (a `Core.CronJobs` row;
   CronScheduler Lambda matches via croniter every minute, no code wiring beyond the row — but
   the action must be deployed), and the **biweekly self-gating pattern**: for schedules cron

package/knowledge/2.0/apps/worker2/features/talos-transcript-ingestion.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-13
+updated: 2026-07-28
 owners: [jcardinal]
 files:
   - worker2/Worker/Team/Transcripts.php
@@ -30,6 +30,7 @@ related:
   - ./creating-worker-actions.md
   - ../architecture.md
   - ../../_underscore/features/async-query-execution.md
+  - ../../_underscore/features/apirequest-json-content-type.md
   - ../../../1.0/apps/tools/features/talos-kb-documents-admin.md
   - ../../../1.0/apps/test/features/talos-kb-pipeline.md
 ---
@@ -136,6 +137,35 @@ helpers).
    [Background Email-Template Worker](./notification-email-template.md); the template is the
    TOGA Technology `True`-client template).
 
+### `callTalosEndpoint()` — the shared low-level Talos AI caller
+The private static `callTalosEndpoint()` is the single HTTP path used by **both** AI passes
+(`callAICleaningAPI()` = clean+classify, and `buildRecapEmailBody()` = recap JSON). As of
+2026-07-28 it uses **`_ApiRequest`** instead of raw `curl_init`/`curl_exec`, so every outbound
+Talos call is recorded in the api log. Its contract to the two callers is unchanged:
+`array{success: bool, result?: array, error?: string}`.
+
+Three deliberate deviations from the usual worker2 `_ApiRequest` idiom — do not "normalize" them:
+
+1. **No `ENCODE__JSON`.** The payload is `json_encode`d by the caller with
+   `JSON_UNESCAPED_SLASHES|JSON_UNESCAPED_UNICODE` and passed as a **ready JSON string with
+   payloadEncoding left null**; the response is decoded locally with `json_decode($r, true)`.
+   Reasons: (a) `_ApiRequest::execute()` decodes JSON responses with `json_decode($payload)` —
+   **no assoc flag** — returning `stdClass`, while both callers index
+   `$res['result']['knowledge_doc']` / `$res['result']['recap_json']` as **arrays**;
+   (b) `ENCODE__JSON` re-encodes without the unescaped flags, inflating every non-ASCII char of
+   a full meeting transcript to `\uXXXX`. Pre-encoding preserves the wire bytes exactly.
+2. **The local retry loop is kept; `setAutoRetry()` is not used.** `setAutoRetry()` waits a
+   **flat** delay and retries **every** non-2xx. This endpoint needs exponential backoff
+   (`AI_BASE_DELAY_MS * 2^(n-1)`, `AI_MAX_ATTEMPTS = 2`) and must fail fast on 4xx other than
+   429. So the surrounding `for` loop is retained and **one `_ApiRequest` is constructed per
+   attempt** — which also yields one api-log row per attempt, useful for diagnosing Talos
+   flakiness.
+3. **Transport failures arrive as a thrown Exception** from `execute()` (its equivalent of
+   `curl_error()`), caught to preserve the same error-array return shape.
+
+Timeouts/attempts are class constants: `AI_DEFAULT_TIMEOUT = 600`, `AI_CONNECT_TIMEOUT = 20`,
+`AI_MAX_ATTEMPTS = 2`, `AI_BASE_DELAY_MS`.
+
 ### Organizer email resolution
 - `resolveOrganizerEmail()` calls Graph `/users/{id}?$select=mail,userPrincipalName` — but
   this needs the **`User.Read.All`** app permission, which the app registration **lacks**.
@@ -240,12 +270,40 @@ part of the ingestion loop** (raw reads removed). Credential values live only in
   `bin/sync-knowledge-bases.php`) to discover `development-team-*` KBs from Bedrock and upsert
   them into `Team.KnowledgeBases`. **Open architectural point:** two competing registries —
   `Team.KnowledgeBases` (worker2) vs `Client_True.VectorIndexes` (Tools UI).
+- **`initialize()` MUST register the Logs DB under `DB_CLIENT_LOGS` or the Talos api-logging is
+  a silent no-op.** `_ApiRequest`'s logging branch writes via `_Model_Client_Logs_Api`, whose
+  `DATABASE` const is `_underscore::DB_CLIENT_LOGS`. `Transcripts::initialize()` previously
+  registered only `DB_TEAM`, so swapping curl → `_ApiRequest` would have produced **zero** log
+  rows. It now also registers `Logs_True` under `DB_CLIENT_LOGS` via `_Config::databaseLogs(...)`
+  (same pattern as `Worker/Team/Sprint.php::initialize()`). The general rule is on
+  [Creating Worker Actions](./creating-worker-actions.md#gotchas) — other workers hitting the
+  same Talos host instead pass `setLogging(false)` deliberately
+  (`Worker/Clickup/Fluffer.php`, `Worker/Vapi.php`, `Worker/Ai/Bdr/Vapi.php`).
+- **KNOWN LIMITATION (deliberately NOT fixed — do not patch it locally): long AI calls can leave
+  api-log rows with null `responseCode`/`responsePayload`.** `_ApiRequest` logs in two halves —
+  it inserts the request row and commits **before** the HTTP call, then `save()`s the response
+  fields **after**. With `AI_DEFAULT_TIMEOUT = 600`, that second save runs on a connection idle
+  for up to 10 minutes, which is exactly the worker2 stale-connection failure mode. Expected
+  symptom: on the slowest transcripts the api-log row has a request but no response. A real fix
+  belongs **inside `_ApiRequest::execute()`** (reconnect/re-register before the post-call save)
+  and affects every framework caller, so it is deferred pending architecture review. Do **not**
+  work around it in `Transcripts.php`.
 - **Cross-ACCOUNT S3.** `togaiq` (us-east-1) approved/archive writes use the `[talos]` key;
   `CopyObject` across accounts is impossible, so archive is get(in-memory)+put. (The old
   toga-private cross-account read is no longer in the loop.)
 
 ## Change history
 
+- 2026-07-28 — **`callTalosEndpoint()` migrated from raw curl to `_ApiRequest`** so both AI
+  passes are recorded in the api log; `initialize()` now also registers `Logs_True` under
+  `_underscore::DB_CLIENT_LOGS` (without it the logging is a silent no-op). Deliberately kept
+  the pre-encoded JSON string (no `ENCODE__JSON` — `execute()` returns `stdClass`, and re-encode
+  loses `JSON_UNESCAPED_SLASHES|JSON_UNESCAPED_UNICODE`) and the local exponential-backoff loop
+  (no `setAutoRetry()` — it is flat-delay and retries every non-2xx). Also fixed an undefined
+  `$data` in the exhausted-retry error message (it was `print_r`'d before assignment; now uses
+  the raw response string) and replaced the hardcoded curl connect timeout with
+  `AI_CONNECT_TIMEOUT = 20`. Caller return contract unchanged, so no caller changed.
+  `php -l` clean; not committed. (jcardinal)
 - 2026-07-13 — Recap email **subject-line date** now displays as `n/j/y` (e.g. `7/9/26`)
   instead of `Y-m-d`, via `DateTime::createFromFormat` with a fallback to the raw
   `$dateSegment`. Cosmetic — `$dateSegment` (S3 filename, metadata sidecar, AI recap body)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.450",
+  "version": "1.0.451",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",