toga-ai 1.0.276 → 1.0.277
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: talos-transcripts
|
|
4
|
+
title: Talos transcript ingestion — cross-account creds, KB registry, admin UI edits
|
|
5
|
+
author: jcardinal
|
|
6
|
+
repos: [worker2, tools, dbchanges2]
|
|
7
|
+
framework: "both"
|
|
8
|
+
client: shared
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-07-07
|
|
11
|
+
updated: 2026-07-07
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: talos-transcripts
|
|
15
|
+
**Date:** 2026-07-07
|
|
16
|
+
**Project/Repo:** worker2 + tools + dbchanges2 (both 1.0 and 2.0)
|
|
17
|
+
**Task:** Harden and extend the Talos meeting-transcript → Bedrock knowledge-base ingestion pipeline (worker2) and its Tools (1.0) "Talos Knowledge Base" admin UI — fix cross-account S3 credentials, populate the KB registry from Bedrock, add move/edit doc features, refine AI classification, and fix a MySQL timeout on long transcripts.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
<!-- Include specific file paths and evidence -->
|
|
23
|
+
- **Cross-account S3 credential split (worker2).** `toga-private` (raw transcripts, us-west-2) and `togaiq` (approved/archive KB bucket, us-east-1) are in DIFFERENT AWS accounts; no single key writes both. Fixed by reading per-bucket credentials: `s3Client()`/`getS3Client()` from `[teams]` (`s3_access_key_id`/`s3_secret_access_key`), `getTalosS3Client()`/`getBedrockClient()` from `[talos]`. Added those keys to `worker2/Config/production.ini`. Reprocessor then succeeded uploading to togaiq. Files: `worker2/Worker/Team/Transcripts.php`, `worker2/Config/production.ini`.
|
|
24
|
+
- **Archive step get+put instead of cross-account CopyObject.** `CopyObject` cannot span accounts — was failing `AccessDenied`. Process() step 8 now reads with `rawS3` and writes with `talosS3`. Confirmed working in reprocessor output.
|
|
25
|
+
- **Tools app togaiq write access.** `App_Talos_S3::client()` now prefers a `[talos]` config section (the togaiq-account key), falling back to `[aws]`, then SDK default chain. Added `[talos]` to `tools/config.production.ini` (and `config.alpha.ini`). Move/delete/edit on the KB browser now work. Files: `tools/_/app/talos/s3.php`, `tools/config.production.ini`, `tools/config.alpha.ini`.
|
|
26
|
+
- **Worker-job enqueue from Tools.** Added `[worker]` (worker2 prod SQS queue url/region) and `[aws]` (main-account key for SQS SendMessage) to `tools/config.production.ini`; `[database_toga2core]` was already present. `App_Worker::enqueue` now queues `SyncKb` re-sync jobs successfully.
|
|
27
|
+
- **KB registry sync from Bedrock.** New action `_Worker_Team_Transcripts::SyncKnowledgeBases(bool $apply=true)` + helper `listAllKnowledgeBases()` list all `development-team-*` Bedrock KBs and upsert into `Team.KnowledgeBases`. New CLI `worker2/bin/sync-knowledge-bases.php --dry-run`. User ran it; table went from 2 rows (general, office-depot) to the full KB set including `elite`. Confirmed via `SELECT slug FROM KnowledgeBases`.
|
|
28
|
+
- **Move-document feature (Tools).** `App_Talos_S3::moveApprovedDocument` (get+put copy of .txt + sidecar, rewrites sidecar `metadataAttributes.client.value.stringValue`, deletes originals) + `copyWithinBucket`/`putObjectBody` helpers; `move` action in post.php; move icon → small centered dialog in get.php. Working end-to-end.
|
|
29
|
+
- **Edit title + contents (Tools).** Pencil icon (text docs only) → edit mode: title text box, Find & Replace tool (ported from `test/team/talos/kb_processor.php`), contents textarea, Save/Cancel top-right. `update` action + `App_Talos_S3::sanitizeTitle`/`updateApprovedDocument` (rewrites body; renames to `YYYY-MM-DD - {title}.txt` preserving date; moves+updates sidecar filename attr; enqueues SyncKb). Backdrop-click disabled while editing. All four files lint clean.
|
|
30
|
+
- **Classification: meeting title now passed to AI.** Process() parses date+title up front and passes `meetingTitle` into `callAICleaningAPI`, surfaced as a `MEETING TITLE:` block; instruction prioritizes it. Fixed short Elite transcripts that were falling to `general`.
|
|
31
|
+
- **Sequential dual re-sync on move.** `SyncKb` now accepts comma-separated slugs and syncs each sequentially with `KB_SYNC_DELAY_SECONDS` spacing; move enqueues one `SyncKb("source,dest")` job. Fixes only-destination-synced race (Bedrock allows one in-flight ingestion per KB).
|
|
32
|
+
- **Last-modified timezone.** `App_Talos_S3::listApproved` converts S3 UTC `LastModified` to `America/Chicago` before formatting `n/j/y g:i A`; sort key stays UTC ISO-8601.
|
|
33
|
+
- **/capture run** published 2 UPDATE docs to `_main` (talos-transcript-ingestion.md, talos-kb-documents-admin.md). Validate OK.
|
|
34
|
+
|
|
35
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
36
|
+
<!-- Exact failure reasons — do not vague-ify -->
|
|
37
|
+
- **Setting env vars `AWS_ACCESS_KEY_ID=... php bin/reprocess-transcripts.php`** to switch credentials: NO EFFECT. The code passes credentials explicitly to the S3Client constructor from `_Config`, which overrides the SDK env-var/default chain. Error stayed `User: arn:aws:iam::502614707982:user/jcardinal ... not authorized`.
|
|
38
|
+
- **Granting the `jcardinal` IAM user S3 perms + using the toga-private-account key for togaiq:** still `AccessDenied` on togaiq. That key belongs to the toga-private account; togaiq is a SEPARATE AWS account. IAM user policy alone can't grant cross-account; and this key simply is not the togaiq key.
|
|
39
|
+
- **Switching move from CopyObject to get+put did NOT fix the tools AccessDenied by itself** — because the tools `[aws]` key was STILL the toga-private-account key, which can read togaiq cross-account but not write. Root cause was the credential, not the operation. (get+put is still retained and correct.)
|
|
40
|
+
- **Two parallel `SyncKb` jobs (one per KB) on move:** only the destination KB actually synced. Bedrock allows one in-flight ingestion per KB and the independent workers raced with no ordering/spacing. Replaced with a single sequential job.
|
|
41
|
+
- **Relying on post-AI `setStatus` writes for long transcripts:** fails with MySQL error 2006 "MySQL server has gone away" — the up-to-600s AI HTTP call leaves the DB connection idle past `wait_timeout`, so every write after it (cleaned/uploaded/archived/synced) dies. Do NOT put status writes after the AI call.
|
|
42
|
+
|
|
43
|
+
## Not tried yet (candidates for next session)
|
|
44
|
+
- **DB reconnect/ping after the AI call** as an alternative that would let `kbSlug`/`approvedKey`/`archiveKey` be persisted to the ledger (currently dropped). User chose the "set status before AI" route instead; reconnect remains an option if ledger detail is wanted.
|
|
45
|
+
- **Rotating the pre-existing leaked credentials** (toga-private + togaiq AWS keys, aegra/talos API key) in `worker2/Config/production.ini` and the old `test/team/talos/kb_processor.ini`. Standing separate track.
|
|
46
|
+
- **Moving the two already-processed Elite transcripts** (currently in `general`/`toga-technology`) into `elite` via the Tools move button, and reprocessing to confirm new classification.
|
|
47
|
+
- **Wiring `[talos]` into `tools/config.production.ini` for other environments** if the browser is used outside production (only production + alpha have it).
|
|
48
|
+
- **Cron already added** for KB registry sync; verify it fires after deploy.
|
|
49
|
+
|
|
50
|
+
## Current file state
|
|
51
|
+
| File | Status | Notes |
|
|
52
|
+
|------|--------|-------|
|
|
53
|
+
| worker2/Worker/Team/Transcripts.php | Modified | Per-bucket S3 creds; archive get+put; SyncKnowledgeBases+listAllKnowledgeBases; SyncKb multi-slug sequential; title→AI; customer-over-company + general-apps→general classification; status set to 'synced' BEFORE AI call, post-AI status writes removed. Lint clean. |
|
|
54
|
+
| worker2/Config/production.ini | Modified | `[teams]` gained s3_access_key_id/secret (toga-private key); `[talos]` gained s3_access_key_id/secret (togaiq key). |
|
|
55
|
+
| worker2/bin/sync-knowledge-bases.php | Created | CLI wrapper for SyncKnowledgeBases; `--dry-run`. |
|
|
56
|
+
| tools/_/app/talos/s3.php | Modified | client() prefers [talos]; copyWithinBucket/putObjectBody; moveApprovedDocument; sanitizeTitle/updateApprovedDocument; listApproved UTC→Central; verbose AWS error messages. |
|
|
57
|
+
| tools/mvc/talos/kb-documents/post.php | Modified | Added move + update actions; move enqueues single SyncKb("src,dest"). |
|
|
58
|
+
| tools/mvc/talos/kb-documents/get.php | Modified | Move dialog + edit modal (title box, Find&Replace, textarea, Save/Cancel). |
|
|
59
|
+
| tools/config.production.ini | Modified | Added [talos], [worker], [aws] sections. |
|
|
60
|
+
| tools/config.alpha.ini | Modified | Added [talos] section. |
|
|
61
|
+
| dbchanges2/Core/2026-07-02a - Teams KnowledgeBases sync cron.sql | Created | Daily 05:00 CT cron for Team/Transcripts/SyncKnowledgeBases. Depends on 2026-05-14 maxExecutionTime migration. |
|
|
62
|
+
| dbchanges2/Team/2026-07-02a - TranscriptPromptTemplate exclude personal notes.sql | Created | REPLACE() adds rule to exclude personal/social content + no "Personal Notes" section. |
|
|
63
|
+
|
|
64
|
+
## Decisions made
|
|
65
|
+
- **Per-bucket credentials, not one shared key.** togaiq and toga-private are separate AWS accounts; `[teams]`=toga-private, `[talos]`=togaiq, `[aws]`=main/SQS. Rejected: single key (impossible cross-account), instance role (prod role lacks togaiq write).
|
|
66
|
+
- **App_Talos_S3 gets its own `[talos]` credentials** rather than changing the app-wide `[aws]` key (which other tools use for us-west-2/toga-private). Precedence: [talos] → [aws] → SDK default.
|
|
67
|
+
- **Bedrock is source of truth for the KB list; `Team.KnowledgeBases` is a cache/allowlist** synced by `SyncKnowledgeBases` (insert-only for new, refresh bedrockKbId for existing, never delete/deactivate) + daily cron.
|
|
68
|
+
- **Single sequential `SyncKb` job for moves**, not two parallel jobs — avoids concurrent Bedrock ingestion per KB and guarantees both sync.
|
|
69
|
+
- **Classification rules (in code `$slugInstruction`, not the DB cleaning prompt):** title-priority; customer-over-company (toga-technology never wins over a real customer); general application/engineering work → `general`, `toga-technology` reserved for company-itself meetings; else `general`.
|
|
70
|
+
- **Prompt edits via surgical `REPLACE()` migration**, preserving `{{PLACEHOLDERS}}` and UI edits, no-op if not found. Rejected: full-text overwrite (would clobber UI edits).
|
|
71
|
+
- **Set ledger status to 'synced' BEFORE the AI call** and drop all post-AI status writes. Accepted trade-off: ledger loses kbSlug/approvedKey/archiveKey detail, and a post-AI failure leaves the row 'synced' (not retried by Scan). Chosen explicitly by the developer over a DB-reconnect approach.
|
|
72
|
+
|
|
73
|
+
## Blockers
|
|
74
|
+
none — all changes are code/config complete and lint clean; remaining work is deploy + verify.
|
|
75
|
+
|
|
76
|
+
## Exact next step
|
|
77
|
+
> Deploy worker2 (Transcripts.php + production.ini) and re-run `ENVIRONMENT=production php bin/reprocess-transcripts.php --limit=1` on a LONG transcript to confirm the MySQL error 2006 ("server has gone away") is gone and the row is marked `synced`. Then apply the two new dbchanges2 migrations (Core KB-sync cron + Team prompt personal-notes exclusion) and deploy the tools changes.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
_Saved by /session-save on 2026-07-07_
|
package/package.json
CHANGED