toga-ai 1.0.481 → 1.0.482

package/knowledge/2.0/apps/ai-bdr/features/vapi-integration.md

@@ -6,8 +6,8 @@ project: AI-BDR
 client: shared
 type: feature
 status: active
-updated: 2026-07-21
-owners: [akhokhani, tcox]
+updated: 2026-07-30
+owners: [akhokhani, tcox, bala]
 files:
   - ai-bdr/vapi/templates/assistant.template.json
   - ai-bdr/vapi/templates/system-prompt.template.md
@@ -122,7 +122,7 @@ Returned in `message.analysis.structuredData` on every `end-of-call-report`:
 | `contactEmail` | string | Email the prospect confirmed; empty if not collected |
 | `emailMatchedKnown` | boolean | True if confirmed email matches `knownEmail` from variableValues |
 | `meetingTime` | ISO 8601 | Present if `callOutcome=meeting_booked` |
-| `callbackTime` | ISO 8601 | Present if `callOutcome=callback_scheduled`; backend converts a spoken time to ISO. **BUG (2026-07-21): the model frequently emits natural language with spelled-out numbers (e.g. `"in ten minutes"`) which the PHP parser cannot handle — the callback is then silently dropped. See the convertDateTime gotcha in [worker2 vapi-webhook-handler.md](../../worker2/features/vapi-webhook-handler.md). Fix candidate on this side: constrain this field's schema to ISO 8601 or a numeric-relative format so the model never spells the number out.** |
+| `callbackTime` | ISO 8601 *(documented; unconstrained free text in practice)* | Present if `callOutcome=callback_scheduled`. **BUG — the single highest-value schema fix (rescoped 2026-07-30).** The PHP side stores this raw and parses it with PHP's relative-date grammar, which returns null for **22 of 32** realistic phrasings — see below. |
 | `callbackContactPreference` | string | `phone` / `email` / `sms` |
 | `callType` | string | Mirrors the `callType` variable injected on this call |
 | `nextCallOpeningContext` | string | One-sentence coaching note for the next attempt's opener |
@@ -135,6 +135,25 @@ To change the schema: `python scripts/update_structured_output.py` (dry
 run by default; pass `--apply` to write). Adding a field needs a paired
 PHP worker handler change — coordinate before deploying.
 
+### `callbackTime` — constraining it here is the recommended durable fix
+
+The worker's `convertDateTime` is broken by the **preposition / filler word**, not (as previously
+recorded) by spelled-out numbers: `"10 minutes"` parses but `"in 10 minutes"` does not;
+`"tomorrow 3pm"` parses but `"tomorrow at 3pm"` does not. `in`, `at`, `around` and `sometime` all
+break it, and `"later"` — which appears verbatim in the worker's own sample payload — is null.
+Every failure means a **promised callback is silently never placed**.
+
+**ISO 8601 parses cleanly on the PHP side today** (`2026-08-01T15:00:00Z` and the `.000Z` variant).
+So: constrain this field's schema to **strict ISO 8601 with offset** and prompt the assistant to
+resolve relative speech to an absolute timestamp before emitting. That removes the entire class of
+failure without touching PHP — which is why it ranks above the PHP-side hardening.
+
+**Unverified (2026-07-30):** the live assistant config could not be inspected from the dev box
+(`C:\WWW\info` is the Next.js funnel — no `vapi/` templates, no `scripts/`, zero matches for
+`callbackTime`). Confirm whether the live schema **already** constrains this field, and which
+server-message types each assistant currently subscribes to, before changing anything. Full
+analysis: [worker2 callback-scheduling.md](../../worker2/features/callback-scheduling.md).
+
 ## Environment variables (`.env.example`)
 
 ```
@@ -181,6 +200,14 @@ ensure UA spoofing is preserved or you'll see 403s with no body.
   `callSummary` over a schema fork.
 
 ## Change history
+- 2026-07-30 — **Rescoped the `callbackTime` bug and promoted the schema-side fix.** The parse
+  failure is caused by the preposition/filler word (`in`, `at`, `around`, `sometime`), **not**
+  spelled-out numbers — 22/32 realistic phrasings return null, `"tomorrow at 3pm"` among them.
+  Since ISO 8601 parses cleanly in PHP, constraining this field to strict ISO 8601 with offset
+  (plus prompting the assistant to resolve relative speech before emitting) is the **recommended
+  durable fix**. Flagged that the live assistant config could not be inspected, so it is unverified
+  whether the schema already constrains the field or which server-message types each assistant
+  subscribes to (this matters — the worker has no `message.type` guard). (tcox)
 - 2026-07-21 — Live go-live findings. Recorded that the healthcare assistant runs
   `firstMessageMode = "assistant-waits-for-user"` (Option B) so a silent answerer gets silence
   (confirmed: 8s mutual-silence call, `endedReason=customer-ended-call`) — testers must speak first.

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -6,6 +6,7 @@
 | [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
 | [All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)](features/all-client-email-queue-monitor.md) | `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0 outbound email queue (`Logs_<Client>.Email` — see [2.0 Email Send Pipeli | worker2/Worker/Monitor/Operations.php, _underscore/Database.php, _underscore/Query.php, dbchanges2/Logs_Client/2026-05-21 - Email.sql |
 | [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
+| [Callback Scheduling ("call me back") — worker2 AI-BDR](features/callback-scheduling.md) | The **AI-BDR callback path**: what happens between a prospect saying *"call me back later"* on a Vapi call and the dialer actually placing that second call. | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php, worker2/Controller/Index.php |
 | [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
 | [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API, replacing a set of native ClickUp automations. | worker2/Worker/Clickup/Design.php, worker2/Worker/Clickup.php, worker2/Controller/ClickupDesignTest.php, _underscore/Component/Api/Clickup/Clickup.php |
 | [ClickUp GitHub-tab Auto-linking & Ticket-id Branch Naming](features/clickup-github-autolink.md) | How ClickUp surfaces branches/PRs/commits in a ticket's **GitHub tab**, and the branch / PR-title naming convention that triggers it. |  |

package/knowledge/2.0/apps/worker2/features/callback-scheduling.md

@@ -0,0 +1,278 @@
+---
+title: Callback Scheduling ("call me back") — worker2 AI-BDR
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-07-30
+owners: [tcox, bala]
+files:
+  - worker2/Worker/Vapi.php
+  - worker2/Worker/Ai/Bdr/Vapi.php
+  - worker2/Controller/Index.php
+related:
+  - vapi-webhook-handler.md
+  - ../../ai-bdr/features/vapi-integration.md
+  - ../../ai-bdr/features/call-orchestration.md
+  - ../../ai-bdr/workflows/safe-call-loop-testing.md
+---
+
+## Summary
+
+The **AI-BDR callback path**: what happens between a prospect saying *"call me back later"* on a
+Vapi call and the dialer actually placing that second call. It spans the webhook
+(`Worker/Vapi.php` — capture, persist, parse, schedule) and the dialer
+(`Worker/Ai/Bdr/Vapi.php` — eligibility, redial).
+
+**It is currently broken in production.** The requested callback is captured and stored
+correctly, but `convertDateTime` cannot parse most of the phrasings prospects actually use, so
+`actionScheduleCallback` aborts one line before the write and **no callback is ever scheduled**.
+This doc records the source-verified root cause, three further defects found on the same path,
+the recommended fix order, and the production queries that confirm it with real data.
+
+> **Status / handoff (2026-07-30).** Diagnosis only — **nothing has been fixed and no worker2
+> code was modified.** tcox investigated (read-only, `_production` @ `ef50345`); **bala is
+> picking the topic up next.** Immediate next step: run **Q1** below against `Client_True` to
+> confirm the root cause with production data, then decide whether the durable fix lands in the
+> **Vapi assistant structured-output schema** (recommended) or in the PHP parser.
+
+## Key files / entry points
+
+| Location | Role |
+|---|---|
+| `Worker/Vapi.php:99, :109–122, :144` | Extracts `callbackTime` from the end-of-call report |
+| `Worker/Vapi.php:161–173, :189–194, :213–215, :256–266` | Persists `c_callbackTime` **raw** + outcome/action onto the attempt |
+| `Worker/Vapi.php:387–431` (`actionScheduleCallback`) | Parses the time and writes `Contacts.dtNextContactRequested` |
+| `Worker/Vapi.php:609–635` (`convertDateTime`) | The parser — **the failure point** |
+| `Worker/Ai/Bdr/Vapi.php:234–241, :314–331` | Dialer eligibility — honours (or blocks) the callback |
+
+## How it works (end to end)
+
+1. **Capture.** The requested time arrives at
+   `message.artifact.structuredOutputs.<schema-uuid>.result.callbackTime`
+   (`Worker/Vapi.php:99` → `:109–122` → `:144`) — **not** in `analysis`, which is always empty.
+   The published schema documents ISO 8601
+   ([vapi-integration.md](../../ai-bdr/features/vapi-integration.md)), but in practice the field
+   is **unconstrained free text**; the file's own sample payload contains
+   `"callbackTime": "later"` (`Worker/Vapi.php:45`).
+2. **Persist (raw).** `updateContactAttempt` stores the string with **no conversion**
+   (`:213–215`) alongside `c_callOutcome='CALLBACK_SCHEDULED'` / `c_actionToTake='SCHEDULE_CALLBACK'`
+   (map at `:161–173`, applied `:189–194`), via `PUT /contact-attempts/{uuid}` (`:256–266`).
+   `c_callbackTime` is `VARCHAR(255)` (dbchanges2 `_modules/Ai/Bdr/2026-03-04 bdrCustomfields.sql:84`),
+   so a malformed value **survives persistence intact** and only dies at the act step.
+3. **Act.** `executePostCallActions` dispatches `SCHEDULE_CALLBACK` → `actionScheduleCallback`,
+   which resolves the campaign timezone (`:400`), calls `self::convertDateTime($callbackTime, …)`
+   (`:401`), and on success writes `Contacts.dtNextContactRequested` (`:414`) and
+   `contactCallTypeId = CALLBACK` (`:417`) via `PUT /contacts/{uuid}` (`:420–430`).
+4. **Redial.** The dialer treats a due `dtNextContactRequested` as the privileged path:
+   it **bypasses call windows, weekly cadence, call-window exception dates and prior DNC**
+   (`Ai/Bdr/Vapi.php:238–303`, `:314–331`). Callbacks also sort first in the per-run queue (`:334`).
+
+## Root cause — `convertDateTime` cannot parse the times prospects actually say
+
+`actionScheduleCallback` aborts when the parser returns null:
+
+- `Worker/Vapi.php:399–406` — logs `[AI-BDR Webhook] Cannot parse callbackTime "…"` and **`return`s
+  early**. `dtNextContactRequested` is never written, `contactCallTypeId` is never flipped, and
+  nothing downstream ever schedules the call.
+- `Worker/Vapi.php:609–635` — the parser, which leans entirely on PHP's relative-date grammar.
+
+**Empirically measured** (parser extracted and executed on PHP 8.1.10): **22 of 32 realistic
+phrasings return null.** Confirmed null: `later`, `in 10 minutes`, `in ten minutes`,
+`tomorrow at 3pm`, `tomorrow at 3`, `Monday at 9am`, `in an hour`, `in 2 days`, `this afternoon`,
+`tomorrow morning`, `end of day`, `around 2pm`, `15 minutes from now`, `next week sometime`.
+
+### The correction — it is the PREPOSITION, not spelled-out numbers
+
+An earlier note on [vapi-webhook-handler.md](vapi-webhook-handler.md) attributed this to
+spelled-out numbers (`"in ten minutes"`). **That is understated and misleading.** The trigger is
+the **preposition / filler word**:
+
+| Parses | Returns null |
+|---|---|
+| `10 minutes` | `in 10 minutes` |
+| `tomorrow 3pm` | `tomorrow at 3pm` |
+| `2026-08-01T15:00:00Z` (and the `.000Z` variant) | `around 2pm`, `next week sometime` |
+
+The words **`in`, `at`, `around`, `sometime`** break PHP's relative-date grammar. Note that
+`"tomorrow at 3pm"` — the exact phrasing the code comment at `:399` claims works — **fails**.
+Spelled-out numbers are a real but secondary contributor. **ISO 8601 parses cleanly**, which is
+why the durable fix belongs in the Vapi schema.
+
+### Ruled out — do not re-investigate the downstream wiring
+
+The "recorded as data but never scheduled as an action" hypothesis is **FALSE**; it fails one
+line *before* the write.
+
+- `actionScheduleCallback` **does** write `dtNextContactRequested` (`:414`) and
+  `contactCallTypeId` (`:417`) via `PUT /contacts/{uuid}` (`:420–430`). An exhaustive grep confirms
+  this is the **only** writer of `dtNextContactRequested` outside the dialer's own IMMEDIATE /
+  403-retry stamps (`Ai/Bdr/Vapi.php:421`, `:477`).
+- The dialer **honours** it, bypassing call windows, cadence, exception dates and prior DNC.
+- **`CALLBACK_SCHEDULED` is NOT a terminal outcome.** The exclusion at
+  `Ai/Bdr/Vapi.php:317–330` lists only `c_dncRequested=1`, `c_actionToTake='DO_NOT_CALL'`, and
+  `c_callOutcome IN ('MEETING_BOOKED','DNC_REQUESTED')` — and the whole block is short-circuited
+  when `dtNextContactRequested IS NOT NULL` (`:316`).
+
+## Defect 2 (COMPLIANCE) — times that *do* parse are frequently wrong, sometimes in the past
+
+There is **no future/sanity check** between the parse and the write (`Worker/Vapi.php:401–414`),
+and dialer eligibility is simply `dtNextContactRequested <= NOW()` (`Ai/Bdr/Vapi.php:241`).
+
+Measured at **2026-07-30 09:29 CT**, campaign timezone `America/New_York`:
+
+| Spoken | Resolved to | Effect |
+|---|---|---|
+| `Thursday` | 2026-07-29 23:00 | **In the past** — redials on the very next cron minute |
+| `tomorrow` | 2026-07-30 23:00 | An **11 PM call** (midnight ET rendered as 23:00 CT) |
+| `3:00 PM tomorrow` | 2026-07-30 23:00 | Time silently dropped — **wrong day AND wrong time** |
+| `2pm` | earlier today | In the past whenever the call ended after 2pm |
+
+Because the callback path **deliberately bypasses `CampaignCallWindows`**, nothing downstream
+catches an 11 PM dial or a redial 60 seconds later. **Treat this as TCPA/compliance exposure, not
+just a UX bug.**
+
+## Defect 3 — `maxAttemptsPerContact` is NOT bypassed for callbacks
+
+The attempt-cap predicate sits **outside** the `dtNextContactRequested` OR-branch
+(`Ai/Bdr/Vapi.php:234–235`; the branch begins at `:238`). Default cap is **3**
+(dbchanges2 `Client/2026-06-03- BLANK_CLIENT_DATABASE.sql:1852`).
+
+So a prospect who asks for a callback **on their 3rd attempt** gets `dtNextContactRequested` set
+correctly and is then **permanently ineligible**. Callbacks bypass call windows, cadence,
+exception dates and prior DNC — but **not** the attempt cap. This is a silent policy conflict:
+the cap overrides an explicit customer request. Decide the policy explicitly (exempt
+callback-requested contacts, or grant +1).
+
+## Defect 4 — the failure is silent all the way out to Vapi
+
+`actionScheduleCallback`'s write (`Worker/Vapi.php:420–430`) is the **only** write in the file
+with **no try/catch** (compare `:255–284` and `:363–367`), and it is called with
+`$throwExceptionOnApiError = true` — `_underscore` `Component/Api/Toga/Toga.php:167` / `:172`
+throw. The controller then **swallows the throw and returns HTTP 200**
+(`worker2/Controller/Index.php:511–526`), so **Vapi never retries** and the callback is lost with
+no external signal. The only trace is `Core.WorkerJobs.failureReason`.
+
+## Smaller defects found on the same path
+
+- **An empty `Campaigns.timezone` silently kills the callback.** `Worker/Vapi.php:400` uses `??`,
+  which does **not** catch `''`. `Campaigns.timezone` is `varchar(64) NOT NULL` with **no default**
+  (blank DB `:1851`), so `''` is storable, `new DateTimeZone('')` throws, and the parse returns
+  null. Verified that `timezone='Central'` also fails — **only IANA identifiers work.**
+- **No audit of what was actually scheduled.** Only the raw string is stored; the resolved
+  datetime is never persisted, so a failed parse cannot be deterministically re-run.
+- **The attempt's call type gets clobbered.** `Worker/Vapi.php:178` writes the **contact's**
+  `contactCallTypeId` onto the **attempt**, overwriting what the dialer stamped at
+  `Ai/Bdr/Vapi.php:701` (and nulls it when the contact's is NULL).
+- **`Contacts.contactCallTypeId` is never reset to SCHEDULED** after a callback completes, so
+  every subsequent call keeps taking the callback prompt branch (`Ai/Bdr/Vapi.php:371–372`).
+- **`Campaigns.dateEnd` kills late callbacks.** `validateCampaignReady:784–786` throws once today
+  > `dateEnd`; it is caught at `ProcessCampaignCalls:144–146` and returned as `success:false`. A
+  callback promised for after the campaign end date **silently never fires**.
+- **Per-run cap of 10** (`MAX_CALLS_PER_MINUTE` at `:8`, applied `:339–340`). Callbacks sort first
+  (`:334`) so this rarely bites, but >10 simultaneously-due callbacks queue.
+
+## Recommended fix direction (decided 2026-07-30 — priority order)
+
+1. **DURABLE FIX — Vapi assistant config side.** Constrain `callbackTime` in the structured-output
+   schema to **strict ISO 8601 with offset**, and prompt the assistant to resolve relative speech
+   to an absolute timestamp before emitting. ISO parses cleanly today, so this removes the whole
+   class of failure. See [vapi-integration.md](../../ai-bdr/features/vapi-integration.md).
+2. **`Worker/Vapi.php:71–79` and `:135`** — add
+   `if (($message->type ?? '') !== 'end-of-call-report') return 'ok';` and stop force-stamping
+   `dtEnded` from `NOW()` (set it only from a real `endedAt`). Independent of the callback bug and
+   arguably **more urgent** — see the concurrency gotcha in
+   [vapi-webhook-handler.md](vapi-webhook-handler.md).
+3. **`:609–635` and `:403–406`** — defensive normalization (strip leading `in `/`at `/`around `/
+   `sometime `, map spelled-out numbers) and, on failure, **schedule a flagged sane default rather
+   than `return`** — never silently drop a promised callback.
+4. **`:401–414`** — clamp the parsed time: if `<= now`, push to the next valid slot; snap into the
+   campaign's call windows; reject beyond ~90 days.
+5. **`Ai/Bdr/Vapi.php:234–235`** — decide the attempt-cap policy explicitly.
+6. **`:420–430`** — add a try/catch matching the file's own pattern and make failures **loud**
+   (Sentry, not just `error_log`).
+7. **`:400`** — use `?:` instead of `??`, validate against `DateTimeZone::listIdentifiers()`, and
+   add a DB default for `Campaigns.timezone`.
+8. **dbchanges2 `_modules/Ai/Bdr/`** — add a resolved `c_callbackTimeResolved DATETIME` column for
+   auditability and re-runnability.
+
+## Diagnostic queries (run against `Client_True`)
+
+**Q1 — decisive.** Join attempts to contacts for callbacks in the last 30 days:
+
+```sql
+SELECT ca.attemptId, ca.contactId, ca.campaignId, ca.dtStarted, ca.dtEnded,
+       ca.c_callOutcome, ca.c_actionToTake, ca.c_callbackTime,
+       c.dtNextContactRequested, c.contactCallTypeId, c.isOkayToCall
+FROM ContactAttempts ca
+JOIN Contacts c ON c.contactId = ca.contactId
+WHERE ca.c_callOutcome = 'CALLBACK_SCHEDULED'
+  AND ca.dtEnded >= DATE_SUB(NOW(), INTERVAL 30 DAY)
+ORDER BY ca.dtEnded DESC;
+```
+
+Interpretation:
+- `c_callbackTime` non-empty **AND** `dtNextContactRequested IS NULL` → **the parse-failure root
+  cause** (or the silent throw of Defect 4).
+- `c_callbackTime` empty/NULL → the model never emitted the field.
+- `dtNextContactRequested < ca.dtEnded` → **scheduled in the past** (Defect 2).
+- Set correctly → scheduling worked; move to Q3 (attempt cap).
+
+**Q2 — what the parser must actually handle:**
+
+```sql
+SELECT c_callbackTime, COUNT(*) AS n
+FROM ContactAttempts
+WHERE c_callbackTime IS NOT NULL AND c_callbackTime <> ''
+GROUP BY c_callbackTime
+ORDER BY n DESC;
+```
+
+**Q3** — contacts with a due `dtNextContactRequested` grouped against `maxAttemptsPerContact`, to
+flag **BLOCKED BY ATTEMPT CAP**.
+**Q4** — open attempts (`dtStarted IS NOT NULL AND dtEnded IS NULL`) with hours-open; these wedge a
+contact out of **all** future dialing.
+**Q5** — active-campaign config sanity: flag blank `timezone` and expired `dateEnd`.
+
+Also:
+- Grep the worker log / Sentry for **`Cannot parse callbackTime`** — **every hit is a lost callback.**
+- In `Core`, surface the silent throw:
+  ```sql
+  SELECT id, dtCompleted, failureReason FROM WorkerJobs
+  WHERE isSuccess = 0 AND failureReason LIKE '%Vapi%'
+  ORDER BY dtCompleted DESC LIMIT 50;
+  ```
+
+## Verification gaps (do not treat these as confirmed)
+
+- **No production DB access from a dev laptop** (the client DB needs VPN) — every production claim
+  above is a **code/schema inference pending Q1–Q5**.
+- **The live Vapi assistant config could not be inspected.** `C:\WWW\info` is the Next.js funnel:
+  no `vapi/` templates, no `scripts/`, zero matches for `callbackTime`. So it is **unconfirmed**
+  whether the live structured-output schema already constrains the field, and which server-message
+  types each assistant currently subscribes to.
+- **Parser results verified on PHP 8.1.10 only** (an `upgrade-to-php8.5` branch exists; PHP's
+  relative-date grammar has been stable across 7.x–8.x, so results are not expected to differ).
+- **Vapi's actual event ordering** — whether a `status-update` can land *after* the
+  `end-of-call-report` — could not be observed.
+- **api2's PUT semantics for an explicit `null` were not confirmed from source.** If api2 ignores
+  an explicit null, the clearing hazard in the webhook doc's concurrency gotcha disappears. Worth a
+  one-line check.
+
+## Change history
+- 2026-07-30 — Initial doc. Read-only root-cause investigation of the production "call me back
+  never happens" bug (worker2 `_production` @ `ef50345`, `_underscore` `_production` @ `5047d135`).
+  Root cause: `convertDateTime` returns null for 22/32 realistic phrasings, so
+  `actionScheduleCallback` aborts before writing `dtNextContactRequested`. **Corrects** the earlier
+  "spelled-out numbers" attribution — the real trigger is the preposition/filler word (`in`, `at`,
+  `around`, `sometime`); `"tomorrow at 3pm"` (the phrasing the code comment claims works) fails,
+  ISO 8601 parses cleanly. Ruled out the downstream wiring (the write, the dialer bypass, and
+  `CALLBACK_SCHEDULED` not being terminal are all fine). Also recorded: parsed times are often
+  wrong/in the past with call windows bypassed (compliance exposure); `maxAttemptsPerContact` is
+  **not** bypassed for callbacks; the failure is silent out to Vapi (HTTP 200, no retry); plus the
+  fix priority order and the Q1–Q5 production queries. Diagnosis only — no code changed; handed off
+  to bala. (tcox)
+</content>
+</invoke>

package/knowledge/2.0/apps/worker2/features/vapi-webhook-handler.md

@@ -6,13 +6,14 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-29
-owners: [snaredla, tcox]
+updated: 2026-07-30
+owners: [snaredla, tcox, bala]
 files:
   - worker2/Worker/Vapi.php
   - worker2/Worker/Ai/Bdr/Vapi.php
   - worker2/Controller/Index.php
 related:
+  - callback-scheduling.md
   - ../../ai-bdr/features/call-orchestration.md
   - ../../ai-bdr/features/vapi-integration.md
   - ../../ai-bdr/features/live-call-status.md
@@ -114,6 +115,30 @@ other types before they reach `processVapiEndOfCallPayload()`.
 - **Symptom:** `Warning: Undefined property: stdClass::$contactAttemptUuid` (promoted to an
   `ErrorException` by the framework error handler, captured by Sentry).
 
+### The guard is STILL ABSENT on `_production` — and it is a concurrency hazard, not a data-quality one
+
+Confirmed 2026-07-30 on `_production` @ `ef50345`: the string `end-of-call-report` appears **only
+in the sample-payload comment at `Worker/Vapi.php:14`**, and a grep for `message->type` across all
+of worker2 returns **zero hits**. `Webhook()` calls `processVapiEndOfCallPayload()`
+**unconditionally** (`:71–79`). Verified consequences:
+
+- **`dtEnded` is force-stamped `NOW()` on EVERY event** (`?: date(…)` at `:135`). So a **mid-call
+  `status-update` closes the attempt while the call is still live**, which releases the
+  in-progress lock at `Ai/Bdr/Vapi.php:305–313` and lets the dialer place **a second concurrent
+  call to the same person, mid-conversation.** `status-update` is live-subscribed on the
+  healthcare assistant ([vapi-integration.md](../../ai-bdr/features/vapi-integration.md)).
+- It also **NULLs `Contacts.dtNextContactRequested` on every event** (`:268–281`).
+- **Ruled out:** it does *not* blank `c_callbackTime` / `c_callOutcome` — the `!== ''` guards at
+  `:189–215` prevent that.
+- **Replay hazard.** Within a single request the order is clear-then-set, so one event is
+  self-consistent. But a late or duplicate event arriving **after** the report re-runs
+  `actionScheduleCallback` against the **raw stored string re-anchored to a new "now"** — so
+  relative times drift forward on each replay, and unparseable ones clear
+  `dtNextContactRequested` **without restoring it**, leaving it permanently NULL.
+
+Fix: `if (($message->type ?? '') !== 'end-of-call-report') return 'ok';` at `:71–79`, and stop
+force-stamping `dtEnded` from `NOW()` — set it only from a real `endedAt`.
+
 ## Inbound / test calls vs worker-placed outbound calls
 
 The metadata shape depends on **who placed the call**, not the event type:
@@ -194,6 +219,13 @@ above is easy to over-read:
   The exclusion (L314–331: `c_dncRequested`, `DO_NOT_CALL`, `MEETING_BOOKED` / `DNC_REQUESTED`
   outcomes) applies only on the non-requested path; a customer who actively re-engages
   (`dtNextContactRequested`) is dialed regardless of a prior terminal outcome.
+- **The attempt cap is the ONE gate the requested path does NOT bypass** (source-verified
+  2026-07-30). The `attemptCount < Campaigns.maxAttemptsPerContact` predicate sits **outside** the
+  `dtNextContactRequested` OR-branch (`Ai/Bdr/Vapi.php:234–235`; the branch starts at `:238`), and
+  the default cap is **3**. So a prospect who asks for a callback on their **3rd** attempt gets
+  `dtNextContactRequested` set correctly and is then **permanently ineligible** — the cap silently
+  overrides an explicit customer request. See
+  [callback-scheduling.md](callback-scheduling.md).
 - **The normal (non-requested) path requires a `CampaignCallWindows` row to EXIST for the current
   day/time.** A campaign with **no** call-window rows can therefore **only ever dial contacts with a
   pending `dtNextContactRequested`** — this is the **safest launch posture for a web-funnel-only
@@ -239,19 +271,30 @@ character — a 37-char value) **passes** the readiness gate and the campaign is
 Vapi `POST /call/phone` **400s** at call time. Validate the assistant identifier's shape before
 assigning an assistant to any campaign.
 
-### Gotcha (HIGH — call quality) — a spoken callback time with a SPELLED-OUT number is silently dropped
+### Gotcha (HIGH — call quality) — most spoken callback times are silently dropped
+
+> **CORRECTED 2026-07-30.** This gotcha previously blamed **spelled-out numbers**
+> (`"in ten minutes"`). That attribution was **understated and misleading** — the real trigger is
+> the **preposition / filler word**, and the failure rate is far higher than the old wording
+> implied (**22 of 32** realistic phrasings return null, measured on PHP 8.1.10).
+
+`actionScheduleCallback` (`Worker/Vapi.php` ~L387–431) parses `structured.callbackTime` through
+`convertDateTime` (~L609–635), which leans entirely on PHP's relative-date grammar. When it
+returns null the method `error_log`s `"Cannot parse callbackTime"` and **`return`s early**
+(`:399–406`) — `dtNextContactRequested` is never set and `callType` is never flipped to
+`CALLBACK`, so the promised callback **never happens** and nothing surfaces the failure.
+
+The words **`in`, `at`, `around`, `sometime`** break the parse: `"10 minutes"` works but
+`"in 10 minutes"` does not; `"tomorrow 3pm"` works but **`"tomorrow at 3pm"` — the exact example
+the code comment at `:399` claims works — does not.** Also null: `later`, `Monday at 9am`,
+`in an hour`, `in 2 days`, `this afternoon`, `tomorrow morning`, `end of day`, `15 minutes from now`.
+**ISO 8601 parses cleanly** (`2026-08-01T15:00:00Z` and the `.000Z` variant), which is why the
+durable fix belongs in the Vapi structured-output schema.
 
-`actionScheduleCallback` (`Worker/Vapi.php` ~L387–431) parses the assistant's
-`structured.callbackTime` through `convertDateTime` (~L609–635), which relies on PHP's date
-parser. PHP handles relative phrasings like `"10 minutes"` and `"tomorrow at 3pm"` — but **NOT
-spelled-out numbers**. When the model emits `c_callbackTime = "in ten minutes"` (natural language,
-number as a word), `convertDateTime` returns **null**, `actionScheduleCallback` `error_log`s
-`"Cannot parse callbackTime"` and **aborts** — `dtNextContactRequested` is never set and `callType`
-is never flipped to `CALLBACK`, so the promised callback **never happens** and nothing surfaces the
-failure. Reproduced live 2026-07-21 (attempt 183, contact 248). Two recommended fixes (do both):
-(a) constrain the Vapi structured-output `callbackTime` schema to ISO 8601 / numeric-relative so the
-model can't spell the number out (see `../../ai-bdr/features/vapi-integration.md`); (b) defensively
-normalize spelled-out numbers (`one`..`sixty`) in `convertDateTime` before parsing.
+**Full root-cause analysis, three further defects on the same path (wrong/past times with call
+windows bypassed; `maxAttemptsPerContact` not bypassed for callbacks; the silent HTTP-200
+failure), the fix priority order, and the production diagnostic queries live in
+[callback-scheduling.md](callback-scheduling.md).**
 
 ### Gotcha (SECURITY) — the Vapi API bearer token is hardcoded in source
 
@@ -287,7 +330,13 @@ status. See the `ai-bdr` skill / `_Model_True_ContactAttempt` for the full field
   `c_vapiCallIdentifier`.
 - **No message-type guard = errors.** Always filter to `end-of-call-report` first.
 - **Persistence is via the Toga REST API**, not a direct model `save()` — failures are
-  caught and logged, not re-thrown.
+  caught and logged, not re-thrown. **One exception:** `actionScheduleCallback`'s
+  `PUT /contacts/{uuid}` (`:420–430`) is the only write in the file **without** a try/catch
+  (compare `:255–284` and `:363–367`) and is called with `$throwExceptionOnApiError = true`
+  (`_underscore Component/Api/Toga/Toga.php:167`, `:172` throw). Because
+  `Controller/Index.php:511–526` swallows the throw and still returns **HTTP 200**, Vapi never
+  retries and the callback is lost with **no external signal** — the only trace is
+  `Core.WorkerJobs.failureReason`.
 - **Timestamps are stored in Central Time**; Cal.com booking converts `c_meetingTime` back to
   UTC.
 
@@ -296,6 +345,20 @@ status. See the `ai-bdr` skill / `_Model_True_ContactAttempt` for the full field
 - [VAPI Integration — assistants, tools, structured output](../../ai-bdr/features/vapi-integration.md)
 
 ## Change history
+- 2026-07-30 — **Corrected** the HIGH callback gotcha: the parse failure is driven by the
+  **preposition/filler word** (`in`, `at`, `around`, `sometime`), **not** spelled-out numbers —
+  `"tomorrow at 3pm"` (the phrasing the code comment claims works) fails, and 22/32 realistic
+  phrasings return null. Split the full root cause + three further defects + fix order + production
+  queries into the new [callback-scheduling.md](callback-scheduling.md). Also, from the same
+  read-only `_production` @ `ef50345` investigation: the **`message.type` guard is still absent**
+  (zero `message->type` hits repo-wide) and is a **concurrency hazard** — `dtEnded` is force-stamped
+  `NOW()` on every event, so a mid-call `status-update` closes the attempt, releases the
+  in-progress lock and lets the dialer place a second concurrent call to the same person; plus a
+  replay hazard that can leave `dtNextContactRequested` permanently NULL. Recorded that the attempt
+  cap is the one gate the requested path does **not** bypass, and that
+  `actionScheduleCallback`'s write is the file's only un-try/catch'd write while the controller
+  still returns HTTP 200 (silent loss, no Vapi retry). Diagnosis only — no code changed; handed off
+  to bala. (tcox)
 - 2026-07-23 — Broadened the SECURITY credential-rotation gotcha: the AI-BDR rotation ticket now
   also covers the **HubSpot access token** and **Toga API client secret** committed to git in the
   old `info` repo's tracked `.env.*` files (treat as compromised), alongside the hardcoded Vapi

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.481",
+  "version": "1.0.482",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",