toga-ai 1.0.260 → 1.0.263
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/tools/INDEX.md +1 -0
- package/knowledge/1.0/apps/tools/features/talos-kb-documents-admin.md +88 -0
- package/knowledge/2.0/apps/worker2/INDEX.md +2 -1
- package/knowledge/2.0/apps/worker2/features/talos-transcript-ingestion.md +158 -0
- package/knowledge/2.0/apps/worker2/features/teams-transcript-export.md +23 -8
- package/knowledge/INDEX.md +2 -2
- package/package.json +1 -1
- package/skills/kickoff/SKILL.md +5 -0
|
@@ -7,5 +7,6 @@
|
|
|
7
7
|
| [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php |
|
|
8
8
|
| [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |
|
|
9
9
|
| [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/initiate/get.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php, tools/mvc/get.php, tools/config.production.ini, tools/config.local.ini |
|
|
10
|
+
| [Talos Knowledge Base Admin UI (KB Documents + Vocabulary)](features/talos-kb-documents-admin.md) | A Tools (1.0) admin UI to browse/fix the Talos knowledge-base documents and manage the transcript-cleanup vocabulary — without a deploy. | tools/mvc/talos/kb-documents/get.php, tools/mvc/talos/kb-documents/post.php, tools/mvc/talos/vocabulary/get.php, tools/mvc/talos/vocabulary/post.php, tools/_/app/talos/s3.php, tools/_/app/worker.php, tools/_/app/nav.php |
|
|
10
11
|
| [Talos Pricing UI (Onboarding, Dashboard, Benchmarks, Cost Factors + Estimator)](features/talos-pricing-ui.md) | The 1.0 (tools app) face of the **Talos Pricing Platform** — a "Talos Pricing" nav folder with four pages plus a client-side estimate engine. | tools/_/app/nav.php, tools/_/app/talos/estimator.php, tools/mvc/talos/onboarding/get.php, tools/mvc/talos/onboarding/post.php, tools/mvc/talos/pricing/get.php, tools/mvc/talos/benchmarks/get.php, tools/mvc/talos/factors/get.php, tools/mvc/talos/factors/post.php, tools/assets/css/style.css |
|
|
11
12
|
| [Deploying Tools to Elastic Beanstalk (PHP 8.5 / Amazon Linux 2023)](workflows/deploy-to-elastic-beanstalk-al2023.md) | How the **Tools** 1.0 app boots on Elastic Beanstalk running `PHP 8.5 on 64bit Amazon Linux 2023/4.13.1 (aarch64)`. | tools/.ebextensions/004_http_to_https.config, tools/.ebextensions/006_mount-s3fs.config, tools/.ebextensions/007_setup_export_cache_folders.config, tools/.ebextensions/008_setup_ldap.config, tools/.ebextensions/009_setup_phpini.config, tools/.ebextensions/020_setup_git_libraries.config, tools/.ebextensions/050_register_instance_to_shared_application_load_balancer.config, tools/ebs/git.json |
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Talos Knowledge Base Admin UI (KB Documents + Vocabulary)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: tools
|
|
5
|
+
project: Tools
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-30
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- tools/mvc/talos/kb-documents/get.php
|
|
13
|
+
- tools/mvc/talos/kb-documents/post.php
|
|
14
|
+
- tools/mvc/talos/vocabulary/get.php
|
|
15
|
+
- tools/mvc/talos/vocabulary/post.php
|
|
16
|
+
- tools/_/app/talos/s3.php
|
|
17
|
+
- tools/_/app/worker.php
|
|
18
|
+
- tools/_/app/nav.php
|
|
19
|
+
related:
|
|
20
|
+
- ./mvc-data-access-patterns.md
|
|
21
|
+
- ./persona-gated-navigation.md
|
|
22
|
+
- ../architecture.md
|
|
23
|
+
- ../../../2.0/apps/worker2/features/talos-transcript-ingestion.md
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Summary
|
|
27
|
+
|
|
28
|
+
A Tools (1.0) admin UI to browse/fix the Talos knowledge-base documents and manage the
|
|
29
|
+
transcript-cleanup vocabulary — without a deploy. It is the human front-end to the worker2
|
|
30
|
+
[Talos Transcript Ingestion](../../../2.0/apps/worker2/features/talos-transcript-ingestion.md)
|
|
31
|
+
pipeline. A new **"Talos Knowledge Base"** nav group (personas *Development Team* / *TOGa
|
|
32
|
+
Technology*) exposes two pages.
|
|
33
|
+
|
|
34
|
+
## Pages
|
|
35
|
+
|
|
36
|
+
### `/talos/kb-documents` — approved-doc browser
|
|
37
|
+
- KB list derived from the **actual S3 sub-folders** under `development-team/` (title-cased,
|
|
38
|
+
e.g. `office-depot` → "Office Depot") — not from `Team.KnowledgeBases`.
|
|
39
|
+
- Lists approved docs excluding `*.metadata.json`; columns Meeting Date (`n/j/y`) / Title /
|
|
40
|
+
Size / Last modified (`n/j/y g:i A`).
|
|
41
|
+
- Clickable rows open an overlay **modal** with a type-aware preview: text inline; PDF via a
|
|
42
|
+
presigned S3 URL in an iframe; Office files via the Google Docs viewer (presigned URL; MS
|
|
43
|
+
Office viewer noted as an alternative); unsupported → download/delete only.
|
|
44
|
+
- Download + delete are icon buttons in the modal. **Delete enqueues the worker `SyncKb`
|
|
45
|
+
re-sync** so Bedrock reflects the removal.
|
|
46
|
+
|
|
47
|
+
### `/talos/vocabulary` — cleanup vocabulary manager
|
|
48
|
+
Single tabbed page: Prompt Template, Applications, Clients, People, Business Terms, Examples,
|
|
49
|
+
Replacements. (The old standalone `/talos/replacements` page was folded in here and deleted.)
|
|
50
|
+
Classic view → Edit → Save per tab, with **full-set reconciliation** (transactional, rolls
|
|
51
|
+
back on error), trash-icon row removal, and an auto-dismissing toast. Backs the `Team.*`
|
|
52
|
+
prompt/replacement tables that the worker2 pipeline reads at runtime.
|
|
53
|
+
|
|
54
|
+
## Helpers
|
|
55
|
+
|
|
56
|
+
- **`App_Talos_S3`** (`_/app/talos/s3.php`) — S3 client; list KB slugs; list/count approved;
|
|
57
|
+
presigned URL; get/delete; slug sanitize+lowercase; approved-prefix path guard.
|
|
58
|
+
- **`App_Worker::enqueue()`** (`_/app/worker.php`) — inserts a `Core.WorkerJobs` row and sends
|
|
59
|
+
the `{"workerJobId": id}` SQS message, so a 1.0 app can enqueue worker2 jobs.
|
|
60
|
+
|
|
61
|
+
## Deploy prerequisites
|
|
62
|
+
|
|
63
|
+
- `aws/aws-sdk-php` installed in `tools/vendor` (S3 + SQS).
|
|
64
|
+
- New `tools/config.*.ini` sections (none exist yet): `[database_core2]` (writable 2.0 Core,
|
|
65
|
+
dbname `Core`), `[worker]` (`queue_url` / `queue_region`), `[aws]` (production).
|
|
66
|
+
- Apply the dbchanges2 `Team/2026-06-30a..e` + `Core/2026-06-30a` migrations in order.
|
|
67
|
+
- Verify the Bedrock KB region / data-source name against the live account (TODOs in code).
|
|
68
|
+
|
|
69
|
+
## Gotchas
|
|
70
|
+
|
|
71
|
+
- **KB list is derived from S3, not the DB.** The browse UI enumerates live S3 folders under
|
|
72
|
+
`development-team/`; `Team.KnowledgeBases` is not the source of truth for what's shown.
|
|
73
|
+
- **Delete is not just an S3 delete** — it also enqueues a worker `SyncKb` job to re-sync the
|
|
74
|
+
Bedrock data source.
|
|
75
|
+
- No new secret literals introduced; credentials come from the new config sections above.
|
|
76
|
+
|
|
77
|
+
## Change history
|
|
78
|
+
|
|
79
|
+
- 2026-06-30 — Built the Talos KB admin UI: `/talos/kb-documents` (S3-derived KB list, approved
|
|
80
|
+
doc browser with type-aware preview modal, download + delete-with-resync) and `/talos/vocabulary`
|
|
81
|
+
(tabbed prompt/vocabulary/replacements manager with transactional full-set reconciliation;
|
|
82
|
+
folded in and deleted the old `/talos/replacements`). Added `App_Talos_S3` and
|
|
83
|
+
`App_Worker::enqueue()` (1.0 → worker2 SQS) helpers and the "Talos Knowledge Base" nav group. (jcardinal)
|
|
84
|
+
|
|
85
|
+
## Related docs
|
|
86
|
+
|
|
87
|
+
- [Talos Transcript Ingestion Pipeline](../../../2.0/apps/worker2/features/talos-transcript-ingestion.md) — the worker2 backend.
|
|
88
|
+
- [Tools MVC — Routing, CSRF & App_Database Access Patterns](./mvc-data-access-patterns.md)
|
|
@@ -21,8 +21,9 @@
|
|
|
21
21
|
| [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
|
|
22
22
|
| [Talos (TOGa IQ) Meeting-Notes Integration & Token Auto-Refresh (consumer)](features/talos-meeting-notes-integration.md) | How a **dev tool / agent consumes Talos (TOGa IQ)** to query the team meeting-notes corpus programmatically. | .claude/skills/plan-ticket/scripts/talos.js |
|
|
23
23
|
| [Talos Pricing Automation (worker2 Cron — AWS Actuals, Calibration, Monthly Report)](features/talos-pricing-automation.md) | The worker2 half of the **Talos Pricing Platform** (see the talos `pricing-cogs-model` and tools `talos-pricing-ui` docs for the other halves). | worker2/Worker/Talos/Pricing.php, worker2/Database/TalosPricingCrons.sql |
|
|
24
|
+
| [Talos Transcript Ingestion Pipeline (worker2 → AWS Bedrock KBs)](features/talos-transcript-ingestion.md) | `_Worker_Team_Transcripts` (in addition to the upstream `Export` action — see [Teams Meeting Transcript Export](./teams-transcript-export.md)) now runs a fully | worker2/Worker/Team/Transcripts.php, worker2/bin/reprocess-transcripts.php, worker2/Config/production.ini, dbchanges2/Team/2026-06-30a, dbchanges2/Team/2026-06-30b, dbchanges2/Team/2026-06-30c, dbchanges2/Team/2026-06-30d, dbchanges2/Team/2026-06-30e, dbchanges2/Core/2026-06-30a |
|
|
24
25
|
| [Team Sprint Management & Reporting](features/team-sprint-management.md) | `_Worker_Team_Sprint` (file `Worker/Team/Sprint.php`) is the engine behind TOGA's internal **development-sprint process and reporting**. | worker2/Worker/Team/Sprint.php |
|
|
25
|
-
| [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers
|
|
26
|
+
| [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers and archives t | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |
|
|
26
27
|
| [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php |
|
|
27
28
|
| [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |
|
|
28
29
|
| [Ticket → ClickUp Pseudocode Planning (Talos-grounded)](workflows/ticket-to-pseudocode-planning.md) | A repeatable procedure for turning a ClickUp ticket into a reviewed, formatted implementation plan posted back to the ticket's `📝 Pseudocode` custom field. | test/@dave/clickup_md2delta.js |
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Talos Transcript Ingestion Pipeline (worker2 → AWS Bedrock KBs)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-30
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Team/Transcripts.php
|
|
13
|
+
- worker2/bin/reprocess-transcripts.php
|
|
14
|
+
- worker2/Config/production.ini
|
|
15
|
+
- dbchanges2/Team/2026-06-30a
|
|
16
|
+
- dbchanges2/Team/2026-06-30b
|
|
17
|
+
- dbchanges2/Team/2026-06-30c
|
|
18
|
+
- dbchanges2/Team/2026-06-30d
|
|
19
|
+
- dbchanges2/Team/2026-06-30e
|
|
20
|
+
- dbchanges2/Core/2026-06-30a
|
|
21
|
+
related:
|
|
22
|
+
- ./teams-transcript-export.md
|
|
23
|
+
- ./creating-worker-actions.md
|
|
24
|
+
- ../architecture.md
|
|
25
|
+
- ../../../1.0/apps/tools/features/talos-kb-documents-admin.md
|
|
26
|
+
- ../../../1.0/apps/test/features/talos-kb-pipeline.md
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## Summary
|
|
30
|
+
|
|
31
|
+
`_Worker_Team_Transcripts` (in addition to the upstream `Export` action —
|
|
32
|
+
see [Teams Meeting Transcript Export](./teams-transcript-export.md)) now runs a fully
|
|
33
|
+
automated, cron-driven pipeline that ingests raw Teams transcripts into the **Talos / TOGa IQ**
|
|
34
|
+
AWS Bedrock knowledge bases. It replaces the old **manual** two-page web tooling
|
|
35
|
+
(`test/team/talos/kb_processor.php` + `kb_processor.ini` — see
|
|
36
|
+
[Talos Knowledge Base Pipeline](../../../1.0/apps/test/features/talos-kb-pipeline.md)); the
|
|
37
|
+
cleanup/classify/approve/archive/sync logic is a port of that script, but driven by cron jobs
|
|
38
|
+
and by **DB-editable** config instead of an `.ini`.
|
|
39
|
+
|
|
40
|
+
Flow: raw VTT at `s3://toga-private/transcripts/{date}/…` → `Scan` enqueues one `Process` job
|
|
41
|
+
per unprocessed key → `Process` AI-cleans + AI-classifies → approved `.txt` + Bedrock metadata
|
|
42
|
+
sidecar written to `s3://togaiq/development-team/{kb_slug}/approved/` → raw archived → Bedrock
|
|
43
|
+
`startIngestionJob` on that KB's data source.
|
|
44
|
+
|
|
45
|
+
## Key files / entry points
|
|
46
|
+
|
|
47
|
+
- `Worker/Team/Transcripts.php` — actions `Scan`, `Process(string $sourceKey)`,
|
|
48
|
+
`SyncKb(string $kbSlug)` (all alongside the existing `Export`).
|
|
49
|
+
- `bin/reprocess-transcripts.php` — standalone CLI backlog reprocessor.
|
|
50
|
+
- `Config/production.ini` `[talos]` section — S3/Bedrock targets (see Configuration).
|
|
51
|
+
- DB config + ledger tables in the `Team.*` schema (see Data model), authored in dbchanges2
|
|
52
|
+
under `Team/2026-06-30a..e` + `Core/2026-06-30a`.
|
|
53
|
+
|
|
54
|
+
## How it works
|
|
55
|
+
|
|
56
|
+
### `Scan` (cron)
|
|
57
|
+
Lists unprocessed raw transcripts under the `toga-private` `transcripts/` prefix (skips any
|
|
58
|
+
`sourceKey` already in the `Team.TranscriptProcessing` ledger) and enqueues one `Process` job
|
|
59
|
+
each via `_Worker::runTask('Team/Transcripts/Process', ['sourceKey' => …])`. Scheduled by a
|
|
60
|
+
`Core.CronJobs` row (weekday business hours).
|
|
61
|
+
|
|
62
|
+
### `Process(string $sourceKey)`
|
|
63
|
+
Port of `kb_processor`, per transcript:
|
|
64
|
+
1. Fetch the raw VTT from S3 (`toga-private`, **us-west-2**).
|
|
65
|
+
2. Call the TOGa IQ AI endpoint `https://api.togaiq.com/api/ai/generate`, model
|
|
66
|
+
`bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0`, with an **extended** `output_schema`
|
|
67
|
+
returning **both** `knowledge_doc` **and** `kb_slug` — the AI picks the single best active KB
|
|
68
|
+
from the supplied list, or `general`. The system prompt is **assembled at runtime** from DB
|
|
69
|
+
tables: a template body with `{{APPLICATIONS}}`/`{{CLIENTS}}`/`{{PEOPLE}}`/`{{BUSINESS_TERMS}}`/`{{EXAMPLES}}`
|
|
70
|
+
placeholders filled from `Team.TranscriptPromptTerms`.
|
|
71
|
+
3. Apply DB-driven text replacements from `Team.TranscriptReplacements` (`:c` case-sensitive,
|
|
72
|
+
`:w` whole-word, `:cw`) via `preg_replace_callback` **so replacement backrefs stay literal**.
|
|
73
|
+
4. Build the approved name `"{YYYY-MM-DD} - {Title Case}.txt"`.
|
|
74
|
+
5. Upload to `s3://togaiq/development-team/{kb_slug}/approved/` (**us-east-1**) plus a
|
|
75
|
+
`{name}.txt.metadata.json` Bedrock sidecar.
|
|
76
|
+
6. Archive the raw VTT to `…/{kb_slug}/archive/{date}/`, then delete the source.
|
|
77
|
+
7. AWS Bedrock `startIngestionJob` on that KB's data source (retry 5×30s on
|
|
78
|
+
`ConflictException` / throttling). **If `kb_slug === 'general'`, sync ALL active KBs.**
|
|
79
|
+
|
|
80
|
+
Idempotency + status are tracked in `Team.TranscriptProcessing`
|
|
81
|
+
(`pending/cleaned/uploaded/archived/synced/failed`).
|
|
82
|
+
|
|
83
|
+
### `SyncKb(string $kbSlug)`
|
|
84
|
+
On-demand single-KB re-sync (used by the Tools delete flow — see
|
|
85
|
+
[Talos KB Documents Admin](../../../1.0/apps/tools/features/talos-kb-documents-admin.md)).
|
|
86
|
+
|
|
87
|
+
### `bin/reprocess-transcripts.php` (CLI backlog reprocessor)
|
|
88
|
+
Standalone CLI that bootstraps `_underscore`, lists existing raw transcripts under the
|
|
89
|
+
`toga-private` `transcripts/` prefix, and runs each through `Process()`. Flags
|
|
90
|
+
`--prefix` / `--limit` / `--dry-run`. Tolerates **both** the old `{slug}/{date}/` layout and the
|
|
91
|
+
new `{date}/` layout. Requires the `ENVIRONMENT` env var, e.g.
|
|
92
|
+
`ENVIRONMENT=production php bin/reprocess-transcripts.php --dry-run`.
|
|
93
|
+
|
|
94
|
+
### Graph download hardening
|
|
95
|
+
The Graph transcript download is hardened against SSRF / token leak: `https`-only, a
|
|
96
|
+
`*.microsoft.com` host allowlist, and `FOLLOWLOCATION` off.
|
|
97
|
+
|
|
98
|
+
## Data model (`Team.*` schema, DB alias core cluster)
|
|
99
|
+
|
|
100
|
+
Authored in dbchanges2 (`Team/2026-06-30a..e`, `Core/2026-06-30a`, all dated 2026-06-30). All
|
|
101
|
+
PKs `INT UNSIGNED` to match the existing Talos Team-table family.
|
|
102
|
+
|
|
103
|
+
- **`Team.TranscriptReplacements`** — `pattern, replacement, isCaseSensitive, isWholeWord, isActive`.
|
|
104
|
+
Seeded ~109 rules from the old `.ini`.
|
|
105
|
+
- **`Team.TranscriptPromptTerms`** — `category` ENUM (`application/client/person/businessTerm/example`),
|
|
106
|
+
`term, isActive, dtCreated, dtUpdated`. Seeded from the `.ini` CORRECTIONS lists. PEOPLE are
|
|
107
|
+
stored as full "First Last" names (only 2 confidently paired; the rest seeded as individual
|
|
108
|
+
tokens, intended to be reconciled via the Tools vocabulary UI).
|
|
109
|
+
- **`Team.TranscriptPromptTemplate`** — `name, bodyText` (w/ placeholders), `instruction, model,
|
|
110
|
+
temperature, maxTokens, timeoutSeconds, isActive`. One active row seeded from the `.ini` system prompt.
|
|
111
|
+
- **`Team.KnowledgeBases`** — `slug, name, isGeneral, bedrockKbId, isActive`. Seeded `general` +
|
|
112
|
+
`office-depot`. NOTE: the Tools browse UI derives the **live** KB list from S3 folders, not this table.
|
|
113
|
+
- **`Team.TranscriptProcessing`** — `sourceKey UNIQUE(255), kbSlug, approvedKey, archiveKey,
|
|
114
|
+
status` ENUM (`pending/cleaned/uploaded/archived/synced/failed`), `aiModel, failureReason,
|
|
115
|
+
dtProcessed`. Pipeline ledger + idempotency.
|
|
116
|
+
- **`Core.CronJobs`** INSERT for action `Team/Transcripts/Scan` (weekday business hours);
|
|
117
|
+
includes `dtCreated = NOW()` since the column has no default, and depends on the
|
|
118
|
+
`maxExecutionTime` column migration.
|
|
119
|
+
|
|
120
|
+
## Configuration
|
|
121
|
+
|
|
122
|
+
`worker2/Config/production.ini` `[talos]`: `s3_bucket=togaiq`, `s3_region=us-east-1`,
|
|
123
|
+
`bedrock_region=us-east-1`, `kb_prefix=development-team-`, `s3_root_path=development-team/`.
|
|
124
|
+
|
|
125
|
+
## Gotchas / known issues
|
|
126
|
+
|
|
127
|
+
- **Cross-region S3.** The raw bucket `toga-private` is **us-west-2** while `togaiq` is
|
|
128
|
+
**us-east-1**. `Process()` uses two S3 clients; the archive step is a **cross-region copy**.
|
|
129
|
+
- **`[talos]` keys were added to `production.ini` only** — the beta/development configs still
|
|
130
|
+
need them before the pipeline runs in those environments.
|
|
131
|
+
- **`general` KB fans out.** A `Process()` classified as `general` (or a `general` sync) triggers
|
|
132
|
+
a sync of **all** active KBs, not one.
|
|
133
|
+
- **Replacements must use `preg_replace_callback`** so replacement text with `$`/`\` backrefs is
|
|
134
|
+
treated literally (a plain `preg_replace` would interpret them).
|
|
135
|
+
- **worker2 had no prior CLI precedent** — the backlog reprocessor bootstraps by
|
|
136
|
+
`chdir($projectRoot)` then `require vendor/autoload.php` + `_underscore.php`, and requires the
|
|
137
|
+
`ENVIRONMENT` env var.
|
|
138
|
+
- **Deploy TODOs left in code:** verify the live Bedrock KB region and data-source name against
|
|
139
|
+
the actual account before first prod run.
|
|
140
|
+
- **Pre-existing plaintext secrets** in `worker2/Config/production.ini` and the reference
|
|
141
|
+
`kb_processor.ini` (AWS / aegra) should be rotated — not introduced by this work; no new secret
|
|
142
|
+
literals were added to any new file or migration.
|
|
143
|
+
|
|
144
|
+
## Change history
|
|
145
|
+
|
|
146
|
+
- 2026-06-30 — Built the automated cron-driven ingestion pipeline (`Scan`/`Process`/`SyncKb`) +
|
|
147
|
+
the `bin/reprocess-transcripts.php` backlog CLI, porting the manual `kb_processor.php`. Added
|
|
148
|
+
the `Team.*` config/ledger schema (DB-editable replacements, prompt terms, prompt template, KB
|
|
149
|
+
list, processing ledger) and the `Scan` cron. Classification moved from title-based to AI (single
|
|
150
|
+
best KB, or `general`). Config vocabulary/replacements are now DB-driven and editable via the
|
|
151
|
+
Tools UI. (jcardinal)
|
|
152
|
+
|
|
153
|
+
## Related docs
|
|
154
|
+
|
|
155
|
+
- [Teams Meeting Transcript Export](./teams-transcript-export.md) — the upstream S3 producer.
|
|
156
|
+
- [Talos KB Documents Admin](../../../1.0/apps/tools/features/talos-kb-documents-admin.md) — the Tools UI.
|
|
157
|
+
- [Talos Knowledge Base Pipeline](../../../1.0/apps/test/features/talos-kb-pipeline.md) — the manual script this ports.
|
|
158
|
+
- [Creating Worker Actions](./creating-worker-actions.md)
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
10
|
-
owners: ["ajean"]
|
|
9
|
+
updated: 2026-06-30
|
|
10
|
+
owners: ["ajean", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Team/Transcripts.php
|
|
13
13
|
- worker2/Config/production.ini
|
|
@@ -16,13 +16,18 @@ files:
|
|
|
16
16
|
related:
|
|
17
17
|
- ../architecture.md
|
|
18
18
|
- ./creating-worker-actions.md
|
|
19
|
+
- ./talos-transcript-ingestion.md
|
|
19
20
|
---
|
|
20
21
|
|
|
21
22
|
## Summary
|
|
22
23
|
|
|
23
24
|
`_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for
|
|
24
|
-
Teams meeting transcripts produced by a set of organizers
|
|
25
|
-
|
|
25
|
+
Teams meeting transcripts produced by a set of organizers and archives the raw WebVTT to S3.
|
|
26
|
+
As of 2026-06-30 it **no longer classifies by client**: raw transcripts land **flat** at
|
|
27
|
+
`s3://toga-private/transcripts/{YYYY-MM-DD}/{HHMM}_{title-slug}_{shortId}.vtt` (no client
|
|
28
|
+
folder). Classification now happens later, by AI, in the downstream ingestion pipeline — see
|
|
29
|
+
[Talos Transcript Ingestion](./talos-transcript-ingestion.md), which is the consumer of this
|
|
30
|
+
S3 output. Scheduled via a single
|
|
26
31
|
`Core.CronJobs` row firing weekdays at 9:30 / 11:30 / 13:30 / 14:30 / 15:30 / 16:30 / 17:30
|
|
27
32
|
Central (`30 9,11,13,14,15,16,17 * * 1-5`).
|
|
28
33
|
|
|
@@ -69,11 +74,15 @@ date. `createdDateTime` is stamped minutes after the actual occurrence, so it da
|
|
|
69
74
|
correctly. `getMeeting()` is therefore used only for the subject. `dtExported` uses
|
|
70
75
|
`UTC_TIMESTAMP()` (not `NOW()`) so all three timestamps are UTC.
|
|
71
76
|
|
|
72
|
-
**Classification
|
|
73
|
-
`Core.Clients`
|
|
77
|
+
**Classification: REMOVED (2026-06-30).** The old title-based classification (substring match
|
|
78
|
+
against `Core.Clients` + `[teamsClientAliases]`) was inaccurate and has been deleted
|
|
79
|
+
(`classify()`, `buildClientIndex()`, `loadClientNames()`, `loadClientAliases()`, along with the
|
|
80
|
+
old `Backfill()`/`moveObject()`). Classification is now done by the AI in the downstream
|
|
81
|
+
ingestion `Process()` step, which picks the single best knowledge base (or `general`).
|
|
74
82
|
|
|
75
|
-
**S3 layout:** `s3://
|
|
76
|
-
(date/time in Central).
|
|
83
|
+
**S3 layout (current):** `s3://toga-private/transcripts/{YYYY-MM-DD}/{HHMM}_{title-slug}_{shortId}.vtt`
|
|
84
|
+
— **flat, no client/general folder** (date/time in Central). The old
|
|
85
|
+
`…/{client-or-general}/{date}/…` layout is history; the downstream reprocessor CLI tolerates both.
|
|
77
86
|
|
|
78
87
|
## Data model
|
|
79
88
|
|
|
@@ -162,6 +171,12 @@ Policy gap; a `400` only on a UPN means the id was never resolved to a GUID. (A
|
|
|
162
171
|
|
|
163
172
|
## Change history
|
|
164
173
|
|
|
174
|
+
- 2026-06-30 — **Removed title-based client classification and the client-folder S3 layout.**
|
|
175
|
+
Raw transcripts now land flat at `transcripts/{date}/…` with no client subfolder; deleted
|
|
176
|
+
`classify()`/`buildClientIndex()`/`loadClientNames()`/`loadClientAliases()` and the old
|
|
177
|
+
`Backfill()`/`moveObject()`. Classification moved downstream to the AI in the new
|
|
178
|
+
[Talos Transcript Ingestion](./talos-transcript-ingestion.md) pipeline, which is now the
|
|
179
|
+
consumer of this S3 output. (jcardinal)
|
|
165
180
|
- 2026-06-18 — Fixed recurring-meeting date misfiling: archive date + `dtMeeting` now come from
|
|
166
181
|
the transcript `createdDateTime`, not the series-anchor `meeting->startDateTime` (worker2 PR
|
|
167
182
|
#82, merged + deployed). Added `Backfill()` and ran it in prod (61/66 objects re-filed, 0
|
package/knowledge/INDEX.md
CHANGED
|
@@ -12,12 +12,12 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
12
12
|
- **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
|
|
13
13
|
- **test** (Test) — 12 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
|
|
14
14
|
- **toga** (TOGa) — 2 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
|
|
15
|
-
- **tools** (Tools) —
|
|
15
|
+
- **tools** (Tools) — 8 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
|
|
16
16
|
|
|
17
17
|
## 2.0 framework
|
|
18
18
|
|
|
19
19
|
- **_underscore** (_Underscore) _(framework core)_ — 21 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
20
|
-
- **worker2** (Worker) —
|
|
20
|
+
- **worker2** (Worker) — 25 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
21
21
|
- **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
22
22
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
23
23
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED
package/skills/kickoff/SKILL.md
CHANGED
|
@@ -246,6 +246,11 @@ valid choice; this skill is opt-in.
|
|
|
246
246
|
|
|
247
247
|
Tell the developer concisely:
|
|
248
248
|
- Which framework(s), repo(s), and client are in scope, and the local path of each repo.
|
|
249
|
+
- **Name the framework core(s) explicitly and state the `app → core` dependency** for every
|
|
250
|
+
chosen app — e.g. "`worker` (1.0) → `library` core", "`api2` (2.0) → `_underscore` core".
|
|
251
|
+
Every repo depends on its framework's core (1.0 → `library`, 2.0 → `_underscore`); preflight
|
|
252
|
+
always includes it in `loadSet`. Do **not** bury the core inside a generic "loaded as
|
|
253
|
+
summaries" list — the developer must see that the core is loaded and which app pulled it in.
|
|
249
254
|
- Which knowledge docs you loaded (by title). Where a repo/standard has **no knowledge yet**,
|
|
250
255
|
say so explicitly: "No knowledge captured yet for X — `capture` will build it as you work."
|
|
251
256
|
- Confirm you're primed and ready for their first task.
|