toga-ai 1.0.155 → 1.0.156
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
| [Developer Generators (password, UUID)](features/dev-generators.md) | Two tiny **1.0 `App_` framework** convenience scripts for everyday developer needs. | test/team/generate_password.php, test/team/uuid.php |
|
|
10
10
|
| [Forecast vs NetSuite Discrepancy Analysis](features/forecast-netsuite-discrepancy-analysis.md) | `team/forecast-netsuite/discrepancy_analysis.php` detects discrepancies between our **Forecast database** and **NetSuite** (the source of truth for all sales da | test/team/forecast-netsuite/discrepancy_analysis.php |
|
|
11
11
|
| [TableView Builder (2.0 TableViews SQL generator)](features/tableview-builder.md) | `team/tableViewBuilder/` generates SQL `INSERT` statements for the **2.0 `TableViews`**, `TableViewFields`, and `TableViewJoins` tables from a plain SQL `SELECT | test/team/tableViewBuilder/TableViewGenerator.php, test/team/tableViewBuilder/index.php, test/team/tableViewBuilder/Instructions.md |
|
|
12
|
+
| [Talos Knowledge Base Pipeline (Uploader + Processor)](features/talos-kb-pipeline.md) | `team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI knowledge bases. | test/team/talos/kb_uploader.php, test/team/talos/kb_processor.php, test/team/talos/kb_processor.ini |
|
|
12
13
|
| [TOGa 2.0 Client Onboarding SQL Generator](features/toga2-client-onboarding-sql.md) | `team/generate_toga2_onboarding_sql.php` generates the SQL needed to **onboard a new client** onto the 2.0 platform. | test/team/generate_toga2_onboarding_sql.php |
|
|
13
14
|
| [TOGa 2.0 User Cross-Client Access SQL Generator](features/toga2-user-cross-client-access-sql.md) | `team/generate_toga2_user_access_sql.php` generates SQL to grant an existing 2.0 user from a **home client** access to a **cross client**. | test/team/generate_toga2_user_access_sql.php |
|
|
14
15
|
| [URL & Domain Markdown Document Builder](features/url-domain-markdown-document.md) | `team/build_url_domain_markdown_document.php` generates a **markdown document of our URLs and domains** by pulling environments and domains from the 2.0 platfor | test/team/build_url_domain_markdown_document.php |
|
|
@@ -6,7 +6,7 @@ project: Test
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
9
|
+
updated: 2026-06-22
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- test/team/
|
|
@@ -20,6 +20,7 @@ related:
|
|
|
20
20
|
- ./features/active-directory-auth-test.md
|
|
21
21
|
- ./features/url-domain-markdown-document.md
|
|
22
22
|
- ./features/dev-generators.md
|
|
23
|
+
- ./features/talos-kb-pipeline.md
|
|
23
24
|
---
|
|
24
25
|
|
|
25
26
|
## Summary
|
|
@@ -81,9 +82,12 @@ manually against the target database; they do not execute changes themselves.
|
|
|
81
82
|
| `generate_toga2_onboarding_sql.php` | 1.0 | 2.0 client DBs | [toga2-client-onboarding-sql](./features/toga2-client-onboarding-sql.md) |
|
|
82
83
|
| `generate_toga2_user_access_sql.php` | 1.0 | 2.0 client DBs | [toga2-user-cross-client-access-sql](./features/toga2-user-cross-client-access-sql.md) |
|
|
83
84
|
| `generate_password.php`, `uuid.php` | 1.0 | dev utility | [dev-generators](./features/dev-generators.md) |
|
|
85
|
+
| `talos/kb_uploader.php`, `talos/kb_processor.php` | standalone | S3 + AWS Bedrock KBs | [talos-kb-pipeline](./features/talos-kb-pipeline.md) |
|
|
84
86
|
|
|
85
|
-
**Retired / not documented:** `team/DOA_gitbook/`
|
|
87
|
+
**Retired / not documented:** `team/DOA_gitbook/` is being retired.
|
|
86
88
|
|
|
87
89
|
## Change history
|
|
88
90
|
|
|
89
91
|
- 2026-06-16 — Initial capture of the `team/` folder. Registered repo in `registry.json`.
|
|
92
|
+
- 2026-06-22 — Un-retired `team/DOA_talos/` → `team/talos/`; documented its uploader +
|
|
93
|
+
processor in [talos-kb-pipeline](./features/talos-kb-pipeline.md).
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Talos Knowledge Base Pipeline (Uploader + Processor)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: test
|
|
5
|
+
project: Test
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-22
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- test/team/talos/kb_uploader.php
|
|
13
|
+
- test/team/talos/kb_processor.php
|
|
14
|
+
- test/team/talos/kb_processor.ini
|
|
15
|
+
related:
|
|
16
|
+
- ../architecture.md
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Summary
|
|
20
|
+
|
|
21
|
+
`team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI
|
|
22
|
+
knowledge bases. Content flows through S3 folders and ends up in **AWS Bedrock** knowledge
|
|
23
|
+
bases, which back the internal AI agent that answers developer questions.
|
|
24
|
+
|
|
25
|
+
- **`kb_uploader.php`** — the *intake* page. A user pastes a meeting transcript or uploads a
|
|
26
|
+
document; it lands in the S3 `upload/` folder of a chosen knowledge base.
|
|
27
|
+
- **`kb_processor.php`** — the *review/approve* console. A reviewer cleans transcripts (AI +
|
|
28
|
+
automated text rules), approves files into the `approved/` folder, writes a Bedrock metadata
|
|
29
|
+
sidecar, archives the raw source, and triggers a Bedrock ingestion (sync) job.
|
|
30
|
+
|
|
31
|
+
Both are **standalone PHP pages** — they do **not** bootstrap the 1.0 `App_` framework. They
|
|
32
|
+
load the AWS SDK from `../../vendor/autoload.php` (Composer `aws/aws-sdk-php`) and read all
|
|
33
|
+
settings from a shared `kb_processor.ini`. Target PHP 7.2+.
|
|
34
|
+
|
|
35
|
+
> History: this folder was previously named `DOA_talos` (the `DOA_` prefix marked it as
|
|
36
|
+
> deprecated-but-live). It was un-retired and renamed to `talos` on 2026-06-22.
|
|
37
|
+
|
|
38
|
+
## S3 layout
|
|
39
|
+
|
|
40
|
+
All paths are relative to `S3_BUCKET` (`togaiq`) under `S3_ROOT_PATH` (`development-team/`).
|
|
41
|
+
Each **knowledge base** is a folder one level under the root (e.g. `development-team/<kb>/`),
|
|
42
|
+
discovered by listing common prefixes. Within each KB:
|
|
43
|
+
|
|
44
|
+
| Folder (config key) | Default | Role |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| `S3_UPLOAD_FOLDER` | `upload` | Raw intake — where the uploader drops files |
|
|
47
|
+
| `S3_APPROVED_FOLDER` | `approved` | Reviewed/approved content (+ `.metadata.json` sidecars) — the Bedrock data source |
|
|
48
|
+
| `S3_ARCHIVE_FOLDER` | `archive` | Raw source files after they've been approved |
|
|
49
|
+
|
|
50
|
+
A KB named **`general`** is treated specially: it is never auto-synced on approve/delete;
|
|
51
|
+
instead it is synced in bulk via the "Sync All General" flow.
|
|
52
|
+
|
|
53
|
+
## kb_uploader.php — intake
|
|
54
|
+
|
|
55
|
+
Single page with two POST forms (toggled by a client-side mode switcher):
|
|
56
|
+
|
|
57
|
+
1. **Meeting Transcription** (`submit_meeting`) — fields: date, meeting name, KB, raw
|
|
58
|
+
transcript. The meeting name is slash-sanitized; the transcript's repeated blank lines are
|
|
59
|
+
collapsed; the file is named **`YYYY-MM-DD - {MEETING_NAME}.txt`** and `putObject`'d to
|
|
60
|
+
`<root>/<kb>/upload/`. This filename pattern is what the processor later recognizes as a
|
|
61
|
+
meeting (`isMeetingTranscript()` regex).
|
|
62
|
+
2. **File Upload** (`submit_file`) — fields: KB, document name, file. Extension is validated
|
|
63
|
+
against `ALLOWED_EXTENSIONS`; the file is stored as `{documentName}.{ext}` in the same
|
|
64
|
+
`upload/` folder via `putObject` with `SourceFile`.
|
|
65
|
+
|
|
66
|
+
Knowledge-base options come from `getKnowledgeBases()`. All output is escaped with
|
|
67
|
+
`htmlspecialchars()`.
|
|
68
|
+
|
|
69
|
+
## kb_processor.php — review & approve
|
|
70
|
+
|
|
71
|
+
The page lists everything in every KB's `upload/` folder, split into **meetings** (filename
|
|
72
|
+
matches `^\d{4}-\d{2}-\d{2} - .+\.txt$`) and **documents**. It exposes AJAX actions (POST
|
|
73
|
+
`action=...`, JSON responses) plus an **Approved Files Browser** modal. Key actions:
|
|
74
|
+
|
|
75
|
+
| `action` | Purpose |
|
|
76
|
+
|---|---|
|
|
77
|
+
| `get_file_content` | Return an S3 object's text (transcript editor / file viewer) |
|
|
78
|
+
| `prepare_transcript` | AI clean → automated clean → prepend `date - title` header |
|
|
79
|
+
| `save_to_kb` | Approve a file (see flow below) |
|
|
80
|
+
| `download_file` | Stream any S3 key to the browser as an attachment (`Content-Disposition`) |
|
|
81
|
+
| `delete_file` / `delete_approved_file` | Delete from `upload/` / `approved/` (latter re-syncs the KB) |
|
|
82
|
+
| `list_approved_files` / `count_approved_files` | Populate the Approved Files Browser |
|
|
83
|
+
| `start_sync_all_general` / `sync_next_kb` / `get_sync_progress` | Session-backed batch sync of every KB's `general` data source, polled from the front-end |
|
|
84
|
+
|
|
85
|
+
### Transcript cleaning (`prepare_transcript`)
|
|
86
|
+
|
|
87
|
+
1. **AI cleaning** — `callAICleaningAPI()` POSTs to `AI_API_ENDPOINT` (the TOGa IQ AI service,
|
|
88
|
+
model `AI_MODEL_ID`, Bedrock Claude Haiku 4.5) with an `output_schema` requesting a single
|
|
89
|
+
`knowledge_doc` string. The lengthy `AI_SYSTEM_PROMPT` enforces a strict lossless plain-text
|
|
90
|
+
format (OVERVIEW + topical sections, verbatim code blocks, PascalCase tables / camelCase
|
|
91
|
+
columns). Retries twice with backoff on network errors and HTTP 429/5xx.
|
|
92
|
+
2. **Automated cleaning** — `applyAutomatedCleaning()` strips markdown artifacts (backticks,
|
|
93
|
+
`> ` quoting, `# ` headings), normalizes blank lines around headings/lists, removes
|
|
94
|
+
`:contentReference[...]` markers, and applies the `REPLACEMENTS[...]` map from the ini
|
|
95
|
+
(mishear corrections, e.g. `Talus → Talos`). Replacement keys support `:c` (case-sensitive)
|
|
96
|
+
and `:w` (whole-word) flags.
|
|
97
|
+
3. For meeting files, a `YYYY-MM-DD - Title` header is prepended.
|
|
98
|
+
|
|
99
|
+
### Approve flow (`save_to_kb`)
|
|
100
|
+
|
|
101
|
+
1. Write the approved content to `<root>/<newKb>/approved/<filename>` — uploading the cleaned
|
|
102
|
+
text for meetings, or copying the original S3 object directly for documents.
|
|
103
|
+
2. Write a **Bedrock metadata sidecar** `<filename>.metadata.json`
|
|
104
|
+
(`buildApprovedMetadataJson()`): `metadataAttributes` with `slug`, `client` (the KB name),
|
|
105
|
+
`state=approved`, `approved_at`, `filename`, each `includeForEmbedding: false`.
|
|
106
|
+
3. **Archive** the raw source: move `upload/<file>` → `archive/<file>` (copy + delete).
|
|
107
|
+
4. **Sync** the Bedrock knowledge base unless the KB is `general`:
|
|
108
|
+
`syncKnowledgeBaseDataSource()` resolves the KB by name (`KB_PREFIX + kb`, e.g.
|
|
109
|
+
`development-team-<kb>`), finds the matching data source, and `startIngestionJob()`s it with
|
|
110
|
+
a UUID client token. Retries on `ConflictException` / throttling up to `KB_CONFLICT_MAX_RETRIES`.
|
|
111
|
+
|
|
112
|
+
### Approved Files Browser
|
|
113
|
+
|
|
114
|
+
Modal listing the chosen KB's `approved/` files with size/modified, plus per-row **view**
|
|
115
|
+
(text types open in a viewer via `get_file_content`), **download** (⬇️ — form-POSTs
|
|
116
|
+
`download_file` so the browser saves the file locally), and **delete** (re-syncs the KB).
|
|
117
|
+
Added 2026-06-22: the per-row download button (`downloadApprovedFile()`), reusing the
|
|
118
|
+
existing `download_file` handler.
|
|
119
|
+
|
|
120
|
+
## Configuration (`kb_processor.ini`)
|
|
121
|
+
|
|
122
|
+
Shared by both scripts via `parse_ini_file(..., true)`. Sections: AWS creds/region, S3 bucket
|
|
123
|
+
+ folder names, Bedrock (`KB_PREFIX`, sync delays, conflict-retry tuning), `ALLOWED_EXTENSIONS`,
|
|
124
|
+
AI settings (`AI_API_ENDPOINT`, `AI_MODEL_ID`, `AI_API_KEY`, temperature/tokens/timeout, the
|
|
125
|
+
system prompt), and the `REPLACEMENTS[...]` mishear-correction map.
|
|
126
|
+
|
|
127
|
+
> **⚠️ Security:** `kb_processor.ini` currently stores **live AWS access/secret keys and the AI
|
|
128
|
+
> API key in plaintext, committed to the repo.** These should be rotated and moved out of source
|
|
129
|
+
> (per-developer config or environment) — do not copy these values into other files or docs.
|
|
130
|
+
|
|
131
|
+
## Gotchas
|
|
132
|
+
|
|
133
|
+
- **Not framework code** — no `_.php` / `App_Framework_Sandbox`. Don't assume `App_*` helpers
|
|
134
|
+
are available here; everything goes through the AWS SDK and the ini.
|
|
135
|
+
- **`general` KB is special-cased** everywhere — approve/delete skip its per-file sync; use the
|
|
136
|
+
batch "Sync All General" flow instead.
|
|
137
|
+
- **Filename convention is load-bearing**: the `YYYY-MM-DD - Name.txt` pattern is the only thing
|
|
138
|
+
that distinguishes a meeting (gets AI cleaning) from a document (copied as-is).
|
|
139
|
+
- Batch sync state lives in `$_SESSION['sync_progress']` and is driven by front-end polling
|
|
140
|
+
(`sync_next_kb`), one KB per request with a `KB_SYNC_DELAY_SECONDS` pause between.
|
|
141
|
+
|
|
142
|
+
## Change history
|
|
143
|
+
|
|
144
|
+
- 2026-06-22 — Un-retired the folder (`DOA_talos` → `talos`); added a per-file **Download**
|
|
145
|
+
button to the Approved Files Browser; initial knowledge capture of the uploader + processor.
|
package/knowledge/INDEX.md
CHANGED
|
@@ -10,7 +10,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
10
10
|
- **togaview** (TOGa View) — 6 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
|
|
11
11
|
- **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)
|
|
12
12
|
- **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
|
|
13
|
-
- **test** (Test) —
|
|
13
|
+
- **test** (Test) — 11 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
|
|
14
14
|
|
|
15
15
|
## 2.0 framework
|
|
16
16
|
|
package/package.json
CHANGED