toga-ai 1.0.953 → 1.0.955

package/knowledge/1.0/apps/tools/features/clickup-label-aliases-admin.md

@@ -6,21 +6,24 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-06
 owners: ["ajean"]
 files:
   - tools/mvc/clickup/aliases/get.php
   - tools/mvc/clickup/aliases/post.php
   - tools/_/app/clickup/aliases.php
+  - tools/_/app/clickup/aliases/jobs.php
   - tools/_/app/nav.php
+  - tools/assets/css/style.css
 related:
   - ./mvc-data-access-patterns.md
   - ./errors-curation-console.md
   - ./persona-gated-navigation.md
+  - ./worker2-job-bridge.md
   - ../../../2.0/apps/worker2/features/netsuite-opportunity-client-labels.md
 ---
 
-Human side of the NetSuite→ClickUp Stakeholders/End-Customer labels at `/clickup/aliases`: point an unresolved company name at a ClickUp option in `Team.ClickUpLabelAliases`; open when curating that queue or before saving an alias.
+Human side of the NetSuite→ClickUp Stakeholders/End-Customer labels at `/clickup/aliases`: point an unresolved company name at a ClickUp option in `Team.ClickUpLabelAliases`, refresh the option mirror, and run the label backfill from buttons; open when curating that queue, saving an alias, or changing the Sync card.
 
 ## Summary
 
@@ -34,6 +37,7 @@ The human side of the NetSuite→ClickUp **Stakeholders / End Customer** labels.
 - `tools/mvc/clickup/aliases/get.php` — the page (route `/clickup/aliases`).
 - `tools/mvc/clickup/aliases/post.php` — the save; redirect-after-POST, no JavaScript.
 - `tools/_/app/clickup/aliases.php` — `App_Clickup_Aliases`: `options()`, `optionsUpdatedAt()`, `all()`, `setOption()`. Connection alias `db_team` (config section `[database_team]`), tables `ClickUpLabelAliases` + `ClickUpLabelOptions`.
+- `tools/_/app/clickup/aliases/jobs.php` — `App_Clickup_Aliases_Jobs`: the **Sync with ClickUp** card's queueing, in-progress guard and lock (tools PR #17, fix PR #18).
 - `tools/_/app/nav.php` — listed under **Developers** next to `/errors`, personas `Development Team` / `TOGa Technology`.
 
 Field ids are constants on the class: Stakeholders `ec15edfd-5920-4c12-a1dc-94ede4a62438`, End Customer `67bb359f-fa59-40e1-b94a-0076450bb7c0`. **The two fields are separate option-id spaces.**
@@ -48,8 +52,20 @@ Field ids are constants on the class: Stakeholders `ec15edfd-5920-4c12-a1dc-94ed
 - **Auth + CSRF are explicit:** both files call `App_Auth::requireAuth(['TOGa Technology', 'Development Team'])`, and `post.php` additionally calls `App_Page::validateCrossSiteRequestForgery()` — a non-GET request never reaches `App_FrameworkIndex_Tools::actionHandler()`, so **the framework does not validate CSRF for POSTs**.
 - Storage holds **plain text**; escaping happens exactly once at display. Escaping on the way in would accumulate entities and break equality against the NetSuite name.
 
+### Sync with ClickUp card (two buttons)
+Both queue worker2 actions through `App_Worker::enqueue` — see [worker2-job-bridge](./worker2-job-bridge.md). They replace hand-queuing from SQS.
+- **Refresh options** → `Netsuite/OpportunityStakeholderDigest/Daily` `{dryRun:true}`: republishes `Team.ClickUpLabelOptions`, sends no email.
+- **Run backfill** → `Netsuite/OpportunityStakeholderBackfill/Backfill` `{dryRun, exactOnly:true}`. `exactOnly` is always true from this page. Dry run is the default.
+- **Who can do what (decided by ajean):** refresh + dry run = the page personas (`TOGa Technology`, `Development Team`). **Live backfill = `Development Team` only** — checked on the server (`App_Auth::hasAnyPersona` + const `LIVE_BACKFILL_PERSONAS`); the UI only hides the option. Live also needs a server-checked "I checked the dry-run report" box.
+- **In-progress guard:** a button is blocked while its action is running — rule and why in [worker2-job-bridge](./worker2-job-bridge.md#gotchas). Window = 60 min.
+- **Check + insert run under MySQL `GET_LOCK`**; `RELEASE_LOCK` sits in its own try/catch inside `finally`. Lock name = `'tools-aliases:' . md5($action)` — see the 64-char gotcha below.
+- **Failed SQS send** (enqueue outcome b) → the orphan `WorkerJobs` row is set `isSuccess=0`, so it does not block the button for 60 min.
+- **Errors travel as short codes** (`?jobError=<code>`) mapped to fixed text in `get.php`. Never show free text from the URL — a crafted link could show a fake message (CSO finding).
+
 ## Gotchas
-- **⚠ An option you just created in ClickUp is INVISIBLE here until worker2 republishes the mirror.** The dropdowns are as fresh as the last worker2 run. **Fix: run any digest — it refreshes the mirror in ~2 seconds, and `dryRun` works too**, because on the digest `$dryRun` means "do not send the email", *not* "do not write". This was exactly the confusion after adding 19 new options.
+- **⚠ An option you just created in ClickUp is INVISIBLE here until worker2 republishes the mirror.** The dropdowns are as fresh as the last worker2 run. **Fix: press Refresh options** (a digest `dryRun` — on the digest `$dryRun` means "do not send the email", *not* "do not write"; ~2 seconds).
+- **⚠ `GET_LOCK` names max 64 chars — longer names ERROR (MySQL 4163), they do not truncate.** The first lock name `'tools-clickup-aliases:<action>'` was 66/70 chars, so both buttons always failed with "Not queued. The job could not be queued." php-reviewer and cso both missed it; only real MySQL caught it. Hash the variable part. General rule in [mvc-data-access-patterns](./mvc-data-access-patterns.md#gotchas).
+- **The digest can say "nothing to report" while unresolved alias rows still exist** — e.g. names that now match an option exactly. An empty digest is not an empty queue; read the *Needs an alias* list.
 - **While the mirror is EMPTY, saving is disabled entirely** (the queue still renders). That is a distinct state from a failure, and the notice says so: it fills in on the next worker2 digest or backfill run. `optionsUpdatedAt()` shows the freshness date on the page.
 - **A name with no plausible option cannot be fixed here.** ClickUp's public API cannot create a `labels` option, so someone has to add it in the ClickUp UI first. Those rows still list, ordered by impact.
 - **`?saved=0` is a real outcome, not a no-op** — somebody pressed Save without changing a dropdown, or every change was rejected by the allowlist. `get.php` distinguishes "not a save" from "a save that changed nothing" via `isset($_GET['saved'])`.
@@ -60,4 +76,8 @@ Field ids are constants on the class: Stakeholders `ec15edfd-5920-4c12-a1dc-94ed
 - [Tools MVC — data-access patterns](./mvc-data-access-patterns.md)
 - [Errors curation console](./errors-curation-console.md)
 - [Persona-gated navigation](./persona-gated-navigation.md)
+- [Tools → worker2 job bridge](./worker2-job-bridge.md)
 - [ClickUp Opportunity Client vs End-Customer Labels (worker2)](../../../2.0/apps/worker2/features/netsuite-opportunity-client-labels.md)
+
+## Change history
+- 2026-10-06 — Added the Sync with ClickUp card (Refresh options + Run backfill; live backfill Development Team only), WorkerJobs in-progress guard, and the GET_LOCK 64-char fix (tools PR #17, #18). (ajean)

package/knowledge/1.0/apps/tools/features/mvc-data-access-patterns.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-10-05
+updated: 2026-10-06
 owners: [jcardinal, bala, ajean]
 files:
   - tools/_/app/nav.php
@@ -96,6 +96,9 @@ Use the project's own stripper before logging or emailing a request value: `App_
 ### Every `$_GET`/`$_POST` value is HTML-escaped on arrival — JSON posts break
 `library/app/framework.php` (~line 64) runs every `$_REQUEST`/`$_GET`/`$_POST` value through `App_String::xssProtect()` (htmlspecialchars). JSON posted from page JS arrives as `{"..."}` and `json_decode` returns `NULL`. Undo it **once** with `App_String::xssUnprotect()` before `json_decode`, then validate the decoded values as usual.
 
+### `GET_LOCK` names max 64 chars — longer names ERROR, they do not truncate
+MySQL error 4163 *"User-level lock name ... should not exceed 64 characters"*. Any prefix + action/route/uuid can pass 64 without anyone noticing, and code review does not catch it — only a real MySQL run does. **Hash the variable part:** `'tools-aliases:' . md5($action)` (46 chars). Hit twice: [clickup-label-aliases-admin](./clickup-label-aliases-admin.md) (both buttons dead on prod) and api2 [cross-client retrieval](../../../2.0/apps/api2/features/cross-client-data-retrieval.md). Same limit in 2.0.
+
 ### By-reference row helpers
 `buildArrayOfRows`, `fetchOne`, and `fetchRow` take `$res` **by reference**. Assign the `query()` result to a **variable first**, then pass it — passing the call result inline emits PHP "Only variables should be passed by reference", which **renders to the page** when `display_errors` is on.
 
@@ -160,4 +163,5 @@ set_error_handler(fn(): bool => true); // swallow non-fatal notices/deprecations
 - [1.0 MVC page pattern & app skeleton](../../library/features/mvc-page-pattern-and-app-skeleton.md)
 
 ## Change history
+- 2026-10-06 — Added the `GET_LOCK` 64-char gotcha (hash the variable part). (ajean)
 - 2026-10-05 — Added the request-input xssProtect gotcha (JSON posts decode to NULL). (bala)

package/knowledge/1.0/apps/tools/features/worker2-job-bridge.md

@@ -6,8 +6,8 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-10-05
-owners: [mhammontree, bala]
+updated: 2026-10-06
+owners: [mhammontree, bala, ajean]
 files:
   - tools/_/app/worker.php
 related:
@@ -16,6 +16,7 @@ related:
   - ../../../2.0/apps/worker2/features/creating-worker-actions.md
   - ../../../clients/compass-usa/features/people-file-import-admin-page.md
   - ./mvc-data-access-patterns.md
+  - ./clickup-label-aliases-admin.md
   - ../../../2.0/apps/worker2/architecture.md
   - ../../../2.0/apps/worker2/features/stranded-worker-jobs.md
 ---
@@ -55,6 +56,14 @@ empty result — because the wrapper format is worker2's, not a contract.
 - **`isSuccess = 1` does not mean the work succeeded.** Actions that catch their own errors (e.g. all Etilize actions) return `{success:false, errors:[...]}` and the job is still stamped `isSuccess = 1`. Decide success from the decoded `output` JSON (`success`, `errors`, `fatal`), not the column.
 - **`Core.WorkerJobs.parameters` is a MySQL JSON column.** MySQL re-orders keys and adds spaces, so `parameters = '<json string>'` never matches. Compare with `parameters = CAST('<json>' AS JSON)`. Filter on the index `WorkerJobs_scheduling_IDX (isSuccess, dtQueued, dtStarted)` first.
 - **Guard duplicate enqueues in the DB, not the session.** Tools sessions use the `rediscluster` handler with no session locking, so two concurrent requests both read and write `$_SESSION`. Before enqueuing, reuse a pending identical row (same action + `CAST` parameters, last 2 minutes).
+- **Long or self-chunking job: block while it RUNS, not just a 2-minute duplicate window — two checks.** Read the
+  newest `Core.WorkerJobs` row for the action (`ORDER BY id DESC LIMIT 1` — the table has no index on
+  `action`/`dtCreated`). Running = the row is **unfinished** OR its `output` contains
+  `run CONTINUES`, within a 60 min window. The CONTINUES check is required: worker2 hands off the next
+  chunk with SQS-first `_Worker::runTask`, so the next chunk's row **does not exist until pickup** (a
+  CTO review wrongly said it does). 60 min = the SQS visibility timeout — a failed first delivery is
+  hidden that long, then redelivered. Wrap check + insert in `GET_LOCK`. Reference:
+  [clickup-label-aliases-admin](./clickup-label-aliases-admin.md).
 - **Send only the action's exact parameter names** — the dispatcher spreads them as named args; one extra key fails the job (*Unknown named parameter*). See [creating worker actions](../../../2.0/apps/worker2/features/creating-worker-actions.md).
 - **`isSuccess IS NULL` means two different things** — running, or abandoned. Same trap as
   [stranded worker jobs](../../../2.0/apps/worker2/features/stranded-worker-jobs.md).
@@ -81,6 +90,7 @@ empty result — because the wrapper format is worker2's, not a contract.
 4. Dev-bypass login must use a `Client_True` email (e.g. `@togatech.com`, not `@goagilant.com`).
 
 ## Change history
+- 2026-10-06 — Added the in-progress check for self-chunking jobs (newest row unfinished OR `run CONTINUES`, 60 min window), from the /clickup/aliases Sync card. (ajean)
 - 2026-10-05 — Added: isSuccess=1 can hide a failed action, JSON-column parameters compare, DB-level duplicate-enqueue guard (Etilize Refresh page). (bala)
 - 2026-10-01 — Added the local tools→worker2 test setup and the prod-only `[worker]` config fact (TRUE-82414). (bala)
 - 2026-09-23 — Created from TRUE-82250: documented the three `enqueue()` outcomes, the log-line-wrapped `output`, and the CTO decision that tools never speaks SFTP (phpseclib 1.x sha1-only KEX + credential duplication). (mhammontree)

package/knowledge/2.0/apps/worker2/features/creating-worker-actions.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-05
+updated: 2026-10-06
 owners: [jcardinal, dfranks, mhammontree, tcox, bala, ajean, rgirish]
 files:
   - worker2/Worker/
@@ -372,6 +372,12 @@ Debugging consequence: **no row means either "the caller never queued it" or "th
 and never consumed."** Those have completely different fixes. Distinguish them with the **SQS queue
 metrics**, not the database.
 
+A third case: **SQS returned 200 + MessageId, the tier is healthy, and still no row.** A failed first
+delivery stays hidden for the **3600s visibility timeout**, then is redelivered. Seen 2026-10-05 on a
+hand-sent direct-payload message (resend 15 min later ran). So a successful send is not proof the job
+runs soon — check `WorkerJobs` by `uuid`, and if absent, resend (the `uuid` dedupe stops a double run
+only if you reuse the same `uuid`).
+
 #### Telling "the tier stopped consuming" apart WITHOUT SQS metrics
 
 There is a database-side tell, because cron rows keep arriving from the scheduler even while nothing
@@ -533,6 +539,7 @@ Rules: rename **both sides in one PR and one deploy**; grep every repo for the o
   commit-before-SQS transaction pattern that the worker relies on.
 
 ## Change history
+- 2026-10-06 — Absent-row section: a 200 + MessageId send can still produce no row for up to 1h (failed first delivery hidden by the visibility timeout). (ajean)
 - 2026-10-05 — Rule: `runTask` parameters are ids/uuids only; worker reads with bounded wait; producer catch must `_Error::captureException`. (jcardinal)
 - 2026-10-05 — Added read-host read-after-write gotcha for worker actions (TRUE-82185) (mhammontree)
 - 2026-09-17 — Added the **half-written `WorkerJobs` row** tell: a row with `dtStarted` but no

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-client-labels.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-06
 owners: ["ajean"]
 files:
   - worker2/Worker/Netsuite/Opportunity.php
@@ -17,6 +17,7 @@ files:
   - tools/mvc/clickup/aliases/get.php
   - tools/mvc/clickup/aliases/post.php
   - tools/_/app/clickup/aliases.php
+  - tools/_/app/clickup/aliases/jobs.php
 related:
   - ./netsuite-opportunity-sync.md
   - ./creating-worker-actions.md
@@ -129,7 +130,7 @@ Existing tasks are never reached by the sync (see the claim gotcha below), so a
 
 ### Operational baseline (what a real pass looks like)
 
-~769–771 task-bearing opportunities → **1,542 field slots** (2 per opportunity). Three production passes:
+~769–837 task-bearing opportunities → 2 field slots per opportunity. Production passes:
 
 | Pass | Date | Chunks / wall time | Filled | Notes |
 |---|---|---|---|---|
@@ -137,6 +138,7 @@ Existing tasks are never reached by the sync (see the claim gotcha below), so a
 | 2 | 2026-08-31 | 5 chunks, 15m | 39 | after 21 new ClickUp options; 1,143 already set |
 | 3 | 2026-08-31 | 6 chunks, ~17m | 191 | after 100 more options; unmatched 194 → 2, coverage **89.0%** |
 | 4 | 2026-09-02 | 5 chunks | 26 | after the End Customer re-import; 0 unmatched, 0 fuzzy, coverage **100%** (1,552 of 1,552 slots at 776 opportunities) |
+| 5 | 2026-10-05 | 6 chunks, ~15m | 45 | first run from the Tools buttons; options refreshed (Stakeholders 365→370, End Customer 363→368), 10 aliases assigned; 837 examined, 1,629 already set, 0 unmatched / 0 fuzzy / 0 failed |
 
 Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
 
@@ -146,7 +148,11 @@ Every pass: **0 failures**, and totals always reconcile to `examined × 2`.
 
 ### Operating the queue (things you will need and cannot guess)
 
-**The Tools alias dropdown reads worker2's MIRROR, not live ClickUp.** Options you just created in ClickUp are **invisible** on `/clickup/aliases` until worker2 republishes `Team.ClickUpLabelOptions` (`publishLabelOptions()`), and **saving is disabled entirely** while the mirror is empty. **Any digest run refreshes it in ~2 seconds, including `dryRun`** — on the digest, `$dryRun` means "do not send the email", *not* "do not write". This was the fix when 19 newly-added options could not be seen. Details in [Tools ClickUp label-alias admin](../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md).
+**Run it from the Tools page, not by hand.** `/clickup/aliases` has a *Sync with ClickUp* card: **Refresh options** (digest `dryRun`) and **Run backfill** (dry run default, `exactOnly` always true, live = `Development Team` only). Normal round: refresh options → assign aliases → backfill dry run → live → check dry == live. Details in [Tools ClickUp label-alias admin](../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md).
+
+**The Tools alias dropdown reads worker2's MIRROR, not live ClickUp.** New ClickUp options are **invisible** there until worker2 republishes `Team.ClickUpLabelOptions` (`publishLabelOptions()`); saving is disabled while the mirror is empty. Any digest run refreshes it, including `dryRun` — on the digest, `$dryRun` means "do not send the email", *not* "do not write".
+
+**If you DO hand-send an SQS message, a 200 + MessageId is not proof it will run soon.** 2026-10-05: a direct-payload `{uuid, action, parameters}` for the live backfill got 200 + MessageId and never made a `WorkerJobs` row; a resend 15 min later worked. Check `WorkerJobs` by `uuid` before assuming it runs — see [creating-worker-actions](./creating-worker-actions.md#an-absent-workerjobs-row-is-ambiguous--dont-read-it-as-never-queued).
 
 **Seed the alias queue by hand with `recordUnresolvedAliases()`.** Callable directly as a worker action, which is how to surface names the digest's fuzzy blind spot hides (see the gotcha below — still unfixed):
 
@@ -172,6 +178,7 @@ A one-shot rem+add corrector was scoped and deliberately **not** built.
 - **⚠ THE FUZZY BLIND SPOT — fuzzy-above-floor names are invisible to the digest AND refused by the backfill, so those gaps have no work queue and never fill.** The digest's `collectUnmatched()` skips anything with a non-null option id (`if ($outcome['id'] !== null || !in_array($outcome['reason'], ACTIONABLE_REASONS)) continue;`) — and a **fuzzy hit HAS an id**, so a fuzzy name never reaches the PM email and therefore never gets written to `Team.ClickUpLabelAliases`. Meanwhile the backfill at `exactOnly:true` **refuses** fuzzy. Net effect measured in production: the digest reported **3 outstanding issues while 169 field slots were actually empty** — 167 of them fuzzy skips. Coverage plateaus around 89% and nothing surfaces the gap. Consequence for whoever is clearing the queue: **assigning every alias in the digest does not clear the fuzzy set**, because those names were never queued — verified that `Lee County Sheriff's Office`, `Polk County Sheriff's Office` and `Plant City Police` are all absent from `ClickUpLabelAliases`. **Suggested fix (not implemented):** the digest should report fuzzy-above-floor as its own actionable reason.
 - **Adding ClickUp options creates NEW fuzzy near-misses — the fuzzy count can RISE as gaps are filled.** Adding ~121 options moved names out of *unmatched*, but some landed in *fuzzy, skipped* rather than *filled*: a customer that previously had **no** candidate now has one scoring 85–95%. Observed unmatched **231 → 194 → 2** while fuzzy went **166 → 166 → 167**. So "add more options" is not a complete remedy; past a point the remaining work is **aliases, not options**.
 - **An alias is Pass 0 — ahead of exact AND fuzzy — so a WRONG alias lands on tasks, and seed-if-empty makes it STICKY.** `resolveLabelOption()` (`Opportunity.php` ~L1298) returns confidence `alias` and deliberately bypasses `exactOnly`, which is correct (a human judgement outranks machine guessing) but means a mis-entered alias **will** be written. And because the backfill only fills **empty** fields, no later run corrects it — someone must clear the field in ClickUp by hand. **Operational check worth repeating before every live pass:** audit `ClickUpLabelAliases` for mappings whose **first word differs** from the source name. Doing that on 338 hand-entered aliases surfaced `Hewlett Packard Enterprise → HP` sitting alongside `Hewlett-Packard Enterprise-Health and Life Sciences Team → HPE` — HP Inc. and HPE have been separate companies since 2015.
+- **A "nothing to report" digest does NOT mean the alias queue is empty.** Unresolved rows can remain in `Team.ClickUpLabelAliases` (e.g. names that now match an option). Check the Tools *Needs an alias* list.
 - **The run report lists only the first 50 fuzzy entries.** With 167 fuzzy skips, the other 117 are unreviewable *anywhere* — a real reporting gap, and the reason the production fuzzy set had to be counted by query rather than read off the report.
 - **`splitCustomerRefName()` must handle composite sub-customer numbers.** The original pattern `/^(\d+)\s+(.*)$/` (digits then whitespace) could not parse `"6096:6308 Compass Group"` — the colon broke the match, the number stayed in the name, and normalization produced `"60966308compassgroup"`, which can never match a ClickUp option. **15.4% of `Forecast.Customers` use this composite form** (1,418 of 9,211) and they are precisely the hierarchical customers this feature exists to distinguish. Now `/^(\d+(?::\d+)*)\s+(.*)$/` — still requiring the whitespace so `"3M Company"` is not decapitated.
 - **⚠ `splitCustomerRefName()` strips a NUMERIC prefix only — a NAME hierarchy passes through whole, so AN ALIAS MUST BE KEYED ON THE FULL PATH STRING.** NetSuite sub-customers also arrive as a name path, `" : "`-separated — `"ODP Veyer (B2B) : Compass Group"` — not just the numeric `"6096:6308 Compass Group"` the splitter was written for. The name path survives intact, normalizes to `"odp veyer b2b compass group"`, and can **never** match an option. Consequence when you queue an alias: the full path is the needle `resolveLabelOption()` receives, so an alias entered as just `"Compass Group"` **never fires**. **17 of the 30 populated end-customer rows are this shape.**
@@ -190,3 +197,6 @@ A one-shot rem+add corrector was scoped and deliberately **not** built.
 - [NetSuite supporting-record backfill worker](./netsuite-supporting-record-backfill-worker.md).
 - [Tools ClickUp label-alias admin](../../../../1.0/apps/tools/features/clickup-label-aliases-admin.md) — the human alias queue.
 - [SuiteQL / REST API reference](../../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md) — the List/Record `BUILTIN.DF` distinction.
+
+## Change history
+- 2026-10-06 — Tools Sync card buttons replace hand-queuing; added pass 5 (45 filled, dry == live); hand-sent SQS 200 is not proof of a run; empty digest ≠ empty alias queue. (ajean)

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-05
+updated: 2026-10-06
 owners: ["jcardinal", "mhammontree", "bala", "rgirish"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -509,6 +509,18 @@ against the live SFTP:
   (`rawlist` returned php-type=array, `type=1` for files, and `mtime` as a unix epoch matching the
   filename timestamp.)
 
+Timeout facts (apply the [in-run retry](#suppress-a-one-off-transient-in-the-reporter-not-in-oneuptime-in-run-retry) with these):
+
+- **TCP connect timeout = 3rd constructor arg** `new SFTP($host, $port, $timeout)`, default 10s.
+  `setTimeout()` does NOT change it.
+- **`setTimeout()` is per-read**, applied separately to key exchange, auth reply, channel open, SFTP
+  init, and each readdir packet — one attempt can far exceed one timeout value.
+- Bound retries by an **overall deadline from run start**; split remaining time across steps
+  (per-step = min(step cap, remaining / steps)); stop when that falls below a floor. Instance: Staples
+  (75s deadline, 15s cap, 4 steps, 5s floor).
+- **Never retry a login refusal** — throw a distinct code; retries can lock the account.
+- `disconnect()` in `finally`.
+
 **Symptom when either is wrong:** reads guarded by `isset()`/`??` silently make the aged-file count
 always **0**, so the monitor never alarms and reads as perfectly healthy. Test the count against a folder
 you know has aged files.

package/knowledge/clients/staples/features/oneuptime-sftp-folder-monitors.md

@@ -6,8 +6,8 @@ project: Worker
 client: staples
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-10-06
+owners: ["bala", "jcardinal"]
 files:
   - worker2/Worker/Monitor/Staples.php
   - worker2/Config/beta.ini
@@ -60,6 +60,16 @@ The Staples pipeline runs in the **1.0 `worker`** tier, not worker2; worker2 onl
 - Alarm when **>= 6 aged files** (tunable — can drop to 1 for single-file sensitivity).
 - **5-minute cron cadence** (`*/5 * * * *`), chosen to pair with the OneUptime heartbeat criteria (received-in-8-min + `alarm:OK` recovery; not-received-15-min offline; not-received-10-min degraded), matching the existing Compass monitor convention rather than the source crons' hourly rate. The fast cadence is for a **reliable heartbeat**, not because the folders change that fast — the folder cutoffs above are calibrated against the **hourly 1.0 crons**, per the shared doc's "calibrate against the tier that runs the business logic" rule.
 
+### In-run retry (transient connect failures)
+
+Both actions retry the SFTP connect+list inside one run (shared [in-run retry](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#suppress-a-one-off-transient-in-the-reporter-not-in-oneuptime-in-run-retry) + [phpseclib3 timeout facts](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#sftp-monitoring-phpseclib3)). Added after a one-off `sftp_read_failed` / `fsockopen(): ... Connection timed out` took the inbound monitor offline (2026-10-06).
+
+- `MAX_CONNECT_ATTEMPTS=3`, `RETRY_DELAY_SECONDS=5`, hard `SFTP_DEADLINE_SECONDS=75` from run start.
+- Per-step timeout = min(`SFTP_STEP_TIMEOUT_SECONDS=15`, remaining / `SFTP_STEPS_PER_ATTEMPT=4`); stop retrying below `SFTP_MIN_STEP_TIMEOUT_SECONDS=5`.
+- Login refusal throws `ERROR_CODE_LOGIN_FAILED` — not retried (no account lockout).
+- `$sftp->disconnect()` in `finally`; error push includes `attempts`.
+- Worst case ~75s SFTP + 10s push, under the 120s `maxExecutionTime`.
+
 ## Gotchas
 
 - **phpseclib3 `rawlist` returns arrays, and its SFTP type constants are not loaded** — see the shared doc's [SFTP monitoring gotchas](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#sftp-monitoring-phpseclib3). Getting this wrong silently makes the aged-file count always 0, so the monitor never alarms and looks healthy.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.953",
+  "version": "1.0.955",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",