toga-ai 1.0.59 → 1.0.61
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/_underscore/architecture.md +9 -2
- package/knowledge/2.0/apps/saml/INDEX.md +6 -0
- package/knowledge/2.0/apps/saml/architecture.md +88 -0
- package/knowledge/2.0/apps/saml/features/rate-user-provisioning.md +57 -0
- package/knowledge/INDEX.md +4 -0
- package/knowledge/clients/rate/INDEX.md +6 -0
- package/knowledge/clients/rate/features/saml-sso.md +58 -0
- package/knowledge/clients/rate/profile.md +24 -0
- package/knowledge/clients/tow-foundation/INDEX.md +6 -0
- package/knowledge/clients/tow-foundation/features/receipt-processing.md +130 -0
- package/knowledge/clients/tow-foundation/profile.md +35 -0
- package/knowledge/registry.json +3 -1
- package/package.json +1 -1
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
10
|
-
owners: [jcardinal]
|
|
9
|
+
updated: 2026-06-11
|
|
10
|
+
owners: ["jcardinal", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/_underscore.php
|
|
13
13
|
- _underscore/Loader.php
|
|
@@ -273,3 +273,10 @@ multi-file UI components (`.php`/`.html`/`.css`/`.js`) invoked as `<_ComponentNa
|
|
|
273
273
|
| `_Database` / `_Query` | Connection pool + read/write routing / MySQLi wrapper with caching |
|
|
274
274
|
| `_Route` / `_Component` | URI router / HTML component renderer |
|
|
275
275
|
| `_Config` / `_Environment` / `_Loader` / `_Http` / `_Error` | config, env/debug, autoload, JWT/HTTP, errors |
|
|
276
|
+
|
|
277
|
+
## Gotchas / known issues
|
|
278
|
+
|
|
279
|
+
- **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026, commit `fa7835ed`).** Any code that calls `register()` and then writes to that DB must call `_Database::transactionCommit()` before the request ends — otherwise MySQL silently rolls back all writes when the connection closes. Lazy transactions only materialise on the first write, so read-only callers are unaffected. See `_underscore/Database.php:48`. First discovered when Rate SAML user provisioning silently discarded all new user INSERTs (Jun 2026).
|
|
280
|
+
|
|
281
|
+
## Change history
|
|
282
|
+
- 2026-06-11 — Documented lazy transaction gotcha in `_Database::register()` (rgirish)
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# saml (SAML SSO Gateway) — 2.0 knowledge
|
|
2
|
+
|
|
3
|
+
| Doc | Summary | Files |
|
|
4
|
+
|-----|---------|-------|
|
|
5
|
+
| [saml Architecture](architecture.md) | The `saml` repo is the SAML 2.0 / SSO gateway for all TOGa applications. | saml/index.php, saml/_.php, saml/Controller/Index.php, saml/Config/production.ini, saml/.platform/hooks/prebuild/git.sh, saml/.ebextensions/git.php |
|
|
6
|
+
| [Rate SAML User Provisioning](features/rate-user-provisioning.md) | When a Rate user authenticates via SSO, `_Model_Rate_ClientAuthentication::getAuthenticatedSsoUser()` is called by the saml gateway. | _underscore/Model/Rate/ClientAuthentication.php, _underscore/Model/Rate/User.php, _underscore/Model/Rate/Customer.php, _underscore/Model/Rate/Contact.php |
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: saml Architecture
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: saml
|
|
5
|
+
project: SAML SSO Gateway
|
|
6
|
+
client: shared
|
|
7
|
+
type: architecture
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-11
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- saml/index.php
|
|
13
|
+
- saml/_.php
|
|
14
|
+
- saml/Controller/Index.php
|
|
15
|
+
- saml/Config/production.ini
|
|
16
|
+
- saml/.platform/hooks/prebuild/git.sh
|
|
17
|
+
- saml/.ebextensions/git.php
|
|
18
|
+
related:
|
|
19
|
+
- 2.0/apps/_underscore/architecture.md
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Summary
|
|
23
|
+
|
|
24
|
+
The `saml` repo is the SAML 2.0 / SSO gateway for all TOGa applications. It acts as the
|
|
25
|
+
SAML **Service Provider (SP)** — customer Identity Providers (IdPs: Azure AD, Compass,
|
|
26
|
+
Rate, Prudential, etc.) POST authentication assertions here. The gateway validates the
|
|
27
|
+
assertion, maps the identity to a TOGa user, and redirects the browser back to the
|
|
28
|
+
requesting TOGa app with an encrypted auth handoff payload.
|
|
29
|
+
|
|
30
|
+
Built on the `_underscore` PHP framework. Deployed on AWS Elastic Beanstalk (us-east-1).
|
|
31
|
+
|
|
32
|
+
## Entry point & routing
|
|
33
|
+
|
|
34
|
+
All requests rewrite to `index.php` via `.htaccess`. `_.php` sets
|
|
35
|
+
`DEFAULT_CONTROLLER_METHOD = ['_Controller_Index', 'saml']`. Sub-routes:
|
|
36
|
+
|
|
37
|
+
| Path | Purpose |
|
|
38
|
+
|---|---|
|
|
39
|
+
| `/meta` | SP metadata XML — IdPs consume this to configure the integration |
|
|
40
|
+
| `/acs` | Assertion Consumer Service — receives IdP SAMLResponse + RelayState POST |
|
|
41
|
+
| `/sls` | Advertised in metadata but not implemented |
|
|
42
|
+
| *(default)* | Echoes `TOGaHub SAML SSO receiver` |
|
|
43
|
+
|
|
44
|
+
Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
|
|
45
|
+
|
|
46
|
+
## /acs flow
|
|
47
|
+
|
|
48
|
+
1. **RelayState decryption** — `_String::decryptWithKey()` with `API_SECRET_ACCESS_TOKEN`; retries with `_PREVIOUS` for rotation window support.
|
|
49
|
+
2. **Version + time check** — RelayState JSON: `v` (currently `1`), `time`, `domain` UUID, optional `urlParameters`. Validated against `MAX_TIME_TO_LIVE` (300s).
|
|
50
|
+
3. **Parse SAMLResponse** — base64-decoded, deserialized via LightSaml.
|
|
51
|
+
4. **Status check** — proceeds only on `STATUS_SUCCESS`.
|
|
52
|
+
5. **Assertion decryption** — TOGa SP credential from `_underscore/Assets/ssl/`.
|
|
53
|
+
6. **Resolve client + environment** — `_Model_Core_Domain` by RelayState `domain` UUID → `_Model_Core_Client` + `_Model_Core_Environment`. Dev on real AWS instance downgrades to beta.
|
|
54
|
+
7. **Dynamic DB registration** — SQL joins `Clients`, `Databases`, `DatabaseHosts`, `Environments` for the environment slug. Calls `_Database::register()` + `connect()` for `DB_CLIENT`, `DB_CLIENT_LOGS`, `DB_CLIENT_ARCHIVE`.
|
|
55
|
+
8. **Identity mapping** — `_Model_<ClientIdentifier>_ClientAuthentication::getAuthenticatedSsoUser($assertion)`. Wrapped in try/catch; exceptions sent to Sentry.
|
|
56
|
+
9. **Commit DB writes** — `_Database::transactionCommit()` called after provisioning succeeds (see Gotchas).
|
|
57
|
+
10. **Handoff** — encrypts client UUID + user UUID under `API_SECRET_ACCESS_TOKEN`, base64-encodes `{auth: 'encrypted-user-uuid', payload: {client, user}}`, redirects to `<domain>?saml=<payload>`.
|
|
58
|
+
|
|
59
|
+
### Failure responses
|
|
60
|
+
`Invalid SAMLResponse (Code = 1–4)`, `SAML Authentication Failed`, `User Authentication Failed`.
|
|
61
|
+
|
|
62
|
+
## Per-client SSO extension pattern
|
|
63
|
+
|
|
64
|
+
Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentication.php`.
|
|
65
|
+
Each class extends `_Model_Core_ClientAuthentication` + uses `_Trait_ClientAuthentication`.
|
|
66
|
+
To onboard a new SSO client, add the class in `_underscore` — not in this repo.
|
|
67
|
+
|
|
68
|
+
## Deployment
|
|
69
|
+
|
|
70
|
+
- **EB environment:** `saml-production` (us-east-1). DNS: `saml.togahub.com → saml-production.us-east-1.elasticbeanstalk.com`
|
|
71
|
+
- **`_underscore` pulled at deploy time** via `.ebextensions/git.php` + prebuild hook — clones branch `_production`. Deploy `saml` after `_underscore` merges to pick up framework changes.
|
|
72
|
+
- **CodePipeline:** Source (GitHub `_production`) → ManualApproval → Deploy (EB).
|
|
73
|
+
- **Composer deps** installed at postdeploy via `015_install_composer.sh`.
|
|
74
|
+
|
|
75
|
+
## Dependencies
|
|
76
|
+
|
|
77
|
+
- **`litesaml/lightsaml` ^3.0** — SAML 2.0 parsing, assertion decryption, X509.
|
|
78
|
+
- **`sentry/sentry` ^4.9** — exception reporting, initialized at top of `saml()`.
|
|
79
|
+
|
|
80
|
+
## Gotchas / known issues
|
|
81
|
+
|
|
82
|
+
- **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. The `/acs` handler calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
|
|
83
|
+
- **SAML signature verification is commented out.** The IdP X509 cert verification block is disabled. The only gate is the encrypted RelayState. Re-enabling per-IdP signature verification is a priority security improvement.
|
|
84
|
+
- **SLS not implemented.** `/sls` falls through to the default banner.
|
|
85
|
+
- **`set_exception_handler(null)` at top of `saml()`.** Unhandled exceptions output raw PHP errors. All exceptions from `getAuthenticatedSsoUser()` are caught and sent to Sentry.
|
|
86
|
+
|
|
87
|
+
## Change history
|
|
88
|
+
- 2026-06-11 — Added `transactionCommit()` before redirect; wrapped `getAuthenticatedSsoUser()` in try/catch with Sentry; echo+exit on auth failure (rgirish)
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Rate SAML User Provisioning
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: saml
|
|
5
|
+
project: SAML SSO Gateway
|
|
6
|
+
client: rate
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-11
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Model/Rate/ClientAuthentication.php
|
|
13
|
+
- _underscore/Model/Rate/User.php
|
|
14
|
+
- _underscore/Model/Rate/Customer.php
|
|
15
|
+
- _underscore/Model/Rate/Contact.php
|
|
16
|
+
related:
|
|
17
|
+
- 2.0/apps/saml/architecture.md
|
|
18
|
+
- clients/rate/features/saml-sso.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Summary
|
|
22
|
+
|
|
23
|
+
When a Rate user authenticates via SSO, `_Model_Rate_ClientAuthentication::getAuthenticatedSsoUser()`
|
|
24
|
+
is called by the saml gateway. It either finds the existing user by `borrowerID` or
|
|
25
|
+
provisions a full new user record chain in `Client_Rate`.
|
|
26
|
+
|
|
27
|
+
## How it works
|
|
28
|
+
|
|
29
|
+
1. **Extract attributes** from the SAML assertion — all keys use null-safe access (`?? null`).
|
|
30
|
+
2. **Validate required attributes** — `borrowerID` and `email` must be non-empty. Throws `\Exception('SAML assertion missing required attributes: ...')` if either is missing; caught by saml controller and sent to Sentry.
|
|
31
|
+
3. **Load existing user** — `_Model_Rate_User` searched by `c_borrowerId`. If found, return immediately.
|
|
32
|
+
4. **Provision new user** (if not found):
|
|
33
|
+
- Create `_Model_Rate_Customer` (`c_borrowerId`, `name`)
|
|
34
|
+
- Create `_Model_Rate_Contact` (`firstName`, `lastName`, `customerId`)
|
|
35
|
+
- Create `_Model_Client_ContactEmailAddress` (`contactId`, `name`, `emailAddress`)
|
|
36
|
+
- Update Contact with `primaryContactEmailAddressId`
|
|
37
|
+
- Create `_Model_Rate_User` with all fields; reload by PK (`new _Model_Rate_User($userId)`) to populate `uuid`
|
|
38
|
+
- Create `_Model_Client_Users_Role` with `roleId = 1` (Loan Officer default)
|
|
39
|
+
5. Return User object with `id` and `uuid` populated.
|
|
40
|
+
|
|
41
|
+
## Data model
|
|
42
|
+
|
|
43
|
+
- `Client_Rate.Users` — `c_borrowerId VARCHAR(64)` (UUID from IdP)
|
|
44
|
+
- `Client_Rate.Customers` — `c_borrowerId VARCHAR(128)`, `c_netsuiteInternalCustomerId INT`
|
|
45
|
+
- `Client_Rate.Contacts` — `c_loid VARCHAR(64)`, `c_togaCustomerId INT`
|
|
46
|
+
- `Client_Rate.ContactEmailAddresses` — standard base model
|
|
47
|
+
- `Client_Rate.Users_Roles` — `roleId = 1` hardcoded as Loan Officer default
|
|
48
|
+
|
|
49
|
+
## Gotchas / known issues
|
|
50
|
+
|
|
51
|
+
- **`borrowerID` from Rate's IdP is a UUID string**, not a numeric ID.
|
|
52
|
+
- **`phoneNumber` may not be sent by the IdP** — null-safe, nullable in DB. Do not add to required validation.
|
|
53
|
+
- **`_Database::register()` starts a lazy transaction (since Apr 2 2026).** The saml controller must call `_Database::transactionCommit()` after this method returns — otherwise all INSERTs are silently rolled back on connection close. This was the root cause of new users never persisting. See `_underscore/Database.php:48`.
|
|
54
|
+
- **`load()` scopes only by `c_borrowerId`** — no `clientId` filter. Two users sharing a `borrowerID` causes `load()` to return false (count != 1) and triggers duplicate provisioning.
|
|
55
|
+
|
|
56
|
+
## Change history
|
|
57
|
+
- 2026-06-11 — Added null-safe attribute extraction, required validation for `borrowerID` + `email`, Sentry-caught exceptions (rgirish)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -14,6 +14,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
14
14
|
- **api2** (API) — 1 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
15
15
|
- **dbchanges2** (Database Changes) _(framework core)_ — 1 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
16
16
|
- **toga2-supply** (TOGa Supply) — 2 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
17
|
+
- **saml** (SAML SSO Gateway) — 2 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
|
|
18
|
+
- **toga2-view** (TOGa View Frontend) — 0 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)
|
|
17
19
|
|
|
18
20
|
## Clients
|
|
19
21
|
|
|
@@ -22,4 +24,6 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
22
24
|
- **Elite** (`elite`) → [clients/elite/INDEX.md](clients/elite/INDEX.md)
|
|
23
25
|
- **New York City Department of Education** (`nycdoe`) → [clients/nycdoe/INDEX.md](clients/nycdoe/INDEX.md)
|
|
24
26
|
- **Prudential Financial** (`prudential`) → [clients/prudential/INDEX.md](clients/prudential/INDEX.md)
|
|
27
|
+
- **Rate** (`rate`) → [clients/rate/INDEX.md](clients/rate/INDEX.md)
|
|
28
|
+
- **Tow Foundation** (`tow-foundation`) → [clients/tow-foundation/INDEX.md](clients/tow-foundation/INDEX.md)
|
|
25
29
|
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Client: Rate `rate`
|
|
2
|
+
|
|
3
|
+
| Doc | Framework | Summary | Files |
|
|
4
|
+
|-----|-----------|---------|-------|
|
|
5
|
+
| [Rate SAML SSO](features/saml-sso.md) | 2.0 | Rate uses Azure AD as its IdP (`login.rate.com`). | _underscore/Model/Rate/ClientAuthentication.php, saml/Controller/Index.php, toga2-view/src/hooks/useAuthenticationFlow.ts |
|
|
6
|
+
| [Rate](profile.md) | 2.0 | Rate is a mortgage/lending client. | |
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Rate SAML SSO
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: SAML SSO Gateway
|
|
5
|
+
client: rate
|
|
6
|
+
type: client-feature
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-06-11
|
|
9
|
+
owners: ["rgirish"]
|
|
10
|
+
files:
|
|
11
|
+
- _underscore/Model/Rate/ClientAuthentication.php
|
|
12
|
+
- saml/Controller/Index.php
|
|
13
|
+
- toga2-view/src/hooks/useAuthenticationFlow.ts
|
|
14
|
+
related:
|
|
15
|
+
- 2.0/apps/saml/architecture.md
|
|
16
|
+
- 2.0/apps/saml/features/rate-user-provisioning.md
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Summary
|
|
20
|
+
|
|
21
|
+
Rate uses Azure AD as its IdP (`login.rate.com`). Users sign in at Rate's portal, which
|
|
22
|
+
redirects to the TOGa SAML gateway at `saml.togahub.com/acs`. On success the gateway
|
|
23
|
+
redirects the browser back to the Rate frontend with a `?saml=` encrypted payload.
|
|
24
|
+
|
|
25
|
+
## SAML assertion attributes
|
|
26
|
+
|
|
27
|
+
Rate's Azure AD sends these attributes in the assertion:
|
|
28
|
+
|
|
29
|
+
| Attribute key | Description | Required |
|
|
30
|
+
|---|---|---|
|
|
31
|
+
| `borrowerID` | Borrower UUID — used as `c_borrowerId` to identify/create users | Yes |
|
|
32
|
+
| `email` | User email address | Yes |
|
|
33
|
+
| `firstName` | First name | No (null-safe) |
|
|
34
|
+
| `lastName` | Last name | No (null-safe) |
|
|
35
|
+
| `phoneNumber` | Phone number | No (null-safe) |
|
|
36
|
+
|
|
37
|
+
`borrowerID` is a UUID string (not numeric).
|
|
38
|
+
|
|
39
|
+
## Frontend SAML landing (`useAuthenticationFlow.ts`)
|
|
40
|
+
|
|
41
|
+
1. If `?saml=` is in the URL: strip it immediately via `window.history.replaceState` (prevents stale payload replay on refresh), then call `/v2/auth/encrypted-user-uuid`.
|
|
42
|
+
2. On success: decode `urlParameters` into localStorage, upsert loan officer contact, call `login()`, navigate to `/landing`.
|
|
43
|
+
3. On error: navigate to `/login`.
|
|
44
|
+
|
|
45
|
+
## Loan officer upsert
|
|
46
|
+
|
|
47
|
+
After successful auth, if `urlParameters` contains `loid` + `loname`:
|
|
48
|
+
1. Fetch the user's Contact to get their Customer UUID.
|
|
49
|
+
2. Search `Contacts` by `c_loid` — PUT to update name if found; POST to create with `contactType = 'Loan Officer'` (UUID `d1f67efc-a984-aebc-42f5-5313f44f52ed`) linked to the customer.
|
|
50
|
+
|
|
51
|
+
## Gotchas / known issues
|
|
52
|
+
|
|
53
|
+
- **`?saml=` replay** — the frontend strips the param immediately after processing. Before this fix, refreshing replayed the stale payload → EZ-1 "Invalid User".
|
|
54
|
+
- **New users silently not created (Apr 2 – Jun 11 2026)** — `_Database::register()` auto-starts a lazy transaction; saml gateway never called `transactionCommit()`, so all new user INSERTs were rolled back silently. Fixed by adding `_Database::transactionCommit()` before the redirect in `saml/Controller/Index.php`.
|
|
55
|
+
- **Both saml and _underscore must be deployed together** — saml pulls `_underscore` at EB build time. A fix in `_underscore` requires a saml redeploy to take effect.
|
|
56
|
+
|
|
57
|
+
## Change history
|
|
58
|
+
- 2026-06-11 — Fixed new user creation (missing transactionCommit); fixed ?saml= URL replay; added null-safe attribute extraction and required validation (rgirish)
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Rate"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: SAML SSO Gateway
|
|
5
|
+
client: rate
|
|
6
|
+
type: profile
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-06-11
|
|
9
|
+
owners: ["rgirish"]
|
|
10
|
+
files: []
|
|
11
|
+
related:
|
|
12
|
+
- clients/rate/features/saml-sso.md
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Summary
|
|
16
|
+
|
|
17
|
+
Rate is a mortgage/lending client. Their TOGa integration covers home warranty and home
|
|
18
|
+
tech support products. Users authenticate via SAML SSO from Rate's Azure AD IdP
|
|
19
|
+
(`login.rate.com`). New users are auto-provisioned on first login.
|
|
20
|
+
|
|
21
|
+
- **Client DB:** `Client_Rate`
|
|
22
|
+
- **Client identifier:** `Rate`
|
|
23
|
+
- **Production domains:** `https://homewarranty.rate.com`, `https://hometechsupport.rate.com`
|
|
24
|
+
- **Frontend app:** TOGa View (`toga2-view`)
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Client: Tow Foundation `tow-foundation`
|
|
2
|
+
|
|
3
|
+
| Doc | Framework | Summary | Files |
|
|
4
|
+
|-----|-----------|---------|-------|
|
|
5
|
+
| [Credit Card Receipt Processing](features/receipt-processing.md) | 2.0 | Automated processing of credit card receipts uploaded to SharePoint. | worker2/Worker/Client/TowFoundation.php, worker2/Worker/Client/TowFoundation/ProcessReceipts.php |
|
|
6
|
+
| [Tow Foundation](profile.md) | 2.0 | Tow Foundation is a nonprofit client. | |
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Credit Card Receipt Processing"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: Worker
|
|
5
|
+
client: tow-foundation
|
|
6
|
+
type: client-feature
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-06-11
|
|
9
|
+
owners: ["rgirish"]
|
|
10
|
+
files:
|
|
11
|
+
- worker2/Worker/Client/TowFoundation.php
|
|
12
|
+
- worker2/Worker/Client/TowFoundation/ProcessReceipts.php
|
|
13
|
+
related:
|
|
14
|
+
- clients/tow-foundation/profile.md
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## Summary
|
|
18
|
+
|
|
19
|
+
Automated processing of credit card receipts uploaded to SharePoint. Downloads each
|
|
20
|
+
receipt, extracts structured data via Talos AI, cross-verifies against billing-cycle
|
|
21
|
+
statements, renames the file, archives it per cardholder, generates a QuickBooks-ready
|
|
22
|
+
Excel file per person, uploads it to SharePoint, and sends a summary email.
|
|
23
|
+
|
|
24
|
+
Invocation: `{"action": "Client/TowFoundation/ProcessReceipts/Run", "parameters": {}}`
|
|
25
|
+
|
|
26
|
+
Optional filter parameters: `limit` (int), `year` (string e.g. `"2026"`), `person` (string e.g. `"Brent Peterkin"`).
|
|
27
|
+
|
|
28
|
+
## Key files / entry points
|
|
29
|
+
|
|
30
|
+
- `TowFoundation.php` — base class; holds all email constants and `initialize()` (empty — no client DB)
|
|
31
|
+
- `ProcessReceipts.php` — all logic; abstract class extending `_Worker_Client_TowFoundation`
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
### SharePoint folder structure
|
|
36
|
+
```
|
|
37
|
+
Credit Card Receipts/
|
|
38
|
+
{Person}/
|
|
39
|
+
{Year}/
|
|
40
|
+
{BillingCycleFolder}/ ← receipt files live here (e.g. "Amex ending in 06-03-2026")
|
|
41
|
+
{Person} reports/ ← .xlsx billing statements for cross-verification
|
|
42
|
+
Archive/
|
|
43
|
+
{Person}/ ← successfully processed receipts land here (renamed)
|
|
44
|
+
exception/
|
|
45
|
+
{Person}/ ← failed receipts land here (original name preserved)
|
|
46
|
+
3. QB Excel/ ← generated Excel files uploaded here after each run
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
### Processing flow
|
|
50
|
+
|
|
51
|
+
1. **Auth** — OAuth2 client-credentials token from Microsoft Graph
|
|
52
|
+
2. **Walk** — `walkReceiptsFolder()` enumerates all receipt files and statement Excels
|
|
53
|
+
3. **Filter** — optional `$year` / `$person` / `$limit` applied to the receipt list
|
|
54
|
+
4. **Pre-load statements** — all billing-cycle `.xlsx` files under `reports/` are downloaded
|
|
55
|
+
and parsed upfront via PhpSpreadsheet (auto-detect header row by scanning for
|
|
56
|
+
description/amount keywords)
|
|
57
|
+
5. **Pass 1 — Extract** — for each receipt:
|
|
58
|
+
- Enforce size limit (4 MB images, 10 MB documents)
|
|
59
|
+
- Download file bytes from SharePoint
|
|
60
|
+
- POST to Talos AI `/api/ai/generate` → structured `{vendor_name, invoice_date, total, payment_memo, category, ...}`
|
|
61
|
+
- Cross-verify amount ± $0.01 against parsed statement; if matched, override `payment_memo` with statement description
|
|
62
|
+
- Build base filename (no suffix yet) and store in `$pendingRenames`
|
|
63
|
+
- On extract failure → move to `Archive/exception/{Person}/` immediately
|
|
64
|
+
6. **Collision detection** — count base name occurrences across all pending renames
|
|
65
|
+
7. **Pass 2 — Move** — for each pending rename:
|
|
66
|
+
- If base name appears more than once, assign `_a`, `_b`, `_c`... suffix to **all** colliding files (including the first)
|
|
67
|
+
- PATCH SharePoint to rename + move to `Archive/{Person}/`
|
|
68
|
+
8. **Excel generation** — one `.xlsx` per person (PhpSpreadsheet); columns: Row #, Account Name, QB Vendor, Payment Amount, Date, Payment Method, Payment Memo, QB Description, Class, Category, Payment Account, Ref No.
|
|
69
|
+
9. **SharePoint upload** — each Excel uploaded to `{Person}/{Year}/3. QB Excel/` via Graph API PUT
|
|
70
|
+
10. **Summary email** — sent with Excel files attached; To: Jheanelle, CC: Magdalena, BCC: devteam@togatech.com
|
|
71
|
+
|
|
72
|
+
### Renamed file format
|
|
73
|
+
```
|
|
74
|
+
YYYY.MM.DD Name of Cardholder_VendorName_Amount[_a].ext
|
|
75
|
+
```
|
|
76
|
+
- Vendor name: special chars stripped, spaces → underscores
|
|
77
|
+
- Amount: two decimal places (e.g. `42.50`)
|
|
78
|
+
- Suffix `_a`, `_b`... added when date + person + vendor + amount are all identical (e.g. multiple train tickets same day)
|
|
79
|
+
|
|
80
|
+
### Ref No. (QuickBooks)
|
|
81
|
+
Amex billing cycle runs 3rd-to-3rd. `computeRefNo()` returns the cycle-end date as `MMDDYYYY`.
|
|
82
|
+
- Charge on or before the 3rd → cycle ends on the 3rd of the same month
|
|
83
|
+
- Charge after the 3rd → cycle ends on the 3rd of the next month
|
|
84
|
+
|
|
85
|
+
### Per-person constants
|
|
86
|
+
`CLASS_MAP` and `PAYMENT_ACCOUNT_MAP` in `ProcessReceipts.php` map each cardholder's name
|
|
87
|
+
to their QuickBooks Class and Payment Account strings. Ryan Farrell uses Bank of America
|
|
88
|
+
Mastercard; all others use AMEX Open Credit Card.
|
|
89
|
+
|
|
90
|
+
## Email routing
|
|
91
|
+
|
|
92
|
+
Defined as constants in `TowFoundation.php`:
|
|
93
|
+
|
|
94
|
+
| Constant | Value | Role |
|
|
95
|
+
|---|---|---|
|
|
96
|
+
| `NOTIFY_EMAIL_CLIENT` | `Jheanelle@towfoundation.org` | To |
|
|
97
|
+
| `NOTIFY_EMAIL_CC` | `['Magdalena@towfoundation.org']` | CC |
|
|
98
|
+
| `NOTIFY_EMAIL_BCC` | `['devteam@togatech.com']` | BCC |
|
|
99
|
+
| `NOTIFY_EMAIL_DEV` | `devteam@goagilant.com` | To (always) |
|
|
100
|
+
| `NOTIFY_EMAIL_EXTRA` | jcardinal, ajean, bmorton @goagilant.com | To |
|
|
101
|
+
|
|
102
|
+
Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
103
|
+
|
|
104
|
+
## Gotchas / known issues
|
|
105
|
+
|
|
106
|
+
- **`$year` variable shadowing** — `Run()` uses `$year` as both a filter parameter and a
|
|
107
|
+
loop variable (line ~159: `$year = $receipt['year']`). After the loop `$year` holds the
|
|
108
|
+
last receipt's year, not the original filter value. Harmless now but fragile if the loop
|
|
109
|
+
is restructured.
|
|
110
|
+
- **Two-pass rename is required** — collision detection must see ALL base names before any
|
|
111
|
+
file is moved. A single-pass approach would retroactively need to rename the first file
|
|
112
|
+
after discovering a duplicate, which is not possible once it's already on SharePoint.
|
|
113
|
+
- **`_Loader::setThrowExceptionInAutoloaderIfClassNotFound(false)`** — must wrap all
|
|
114
|
+
PhpSpreadsheet calls. ZipStream is a Composer dependency that triggers the autoloader;
|
|
115
|
+
without this flag, it throws on missing class and aborts Excel generation.
|
|
116
|
+
- **Graph API item-ID move** — `renameAndMoveToArchive()` resolves the target folder to an
|
|
117
|
+
item ID before PATCHing. Using path-based `parentReference` causes HTTP 400 on folder
|
|
118
|
+
names with special characters (spaces, dots). Always resolve to ID first.
|
|
119
|
+
- **`ensureFolderPath()` for "3. QB Excel"** — the folder name contains a dot and space;
|
|
120
|
+
`ensureFolderPath` handles this correctly via `rawurlencode` on each path segment.
|
|
121
|
+
- **Statement matching tolerance** — `matchStatementRow()` uses `abs($row['amount'] - $amount) < 0.01`
|
|
122
|
+
for amount matching, then date tie-break for multiple same-amount rows. Returns first
|
|
123
|
+
candidate if date tie-break also ambiguous.
|
|
124
|
+
- **No client DB** — there is no audit trail in MySQL. All state lives in SharePoint folder
|
|
125
|
+
structure and email. If a run is interrupted mid-way, some receipts may be archived
|
|
126
|
+
without a corresponding Excel row.
|
|
127
|
+
|
|
128
|
+
## Change history
|
|
129
|
+
|
|
130
|
+
- 2026-06-11 — Added CC (Magdalena), BCC (devteam@togatech.com), per-cardholder Archive subfolders, Excel upload to "3. QB Excel" SharePoint folder, two-pass rename with alphabetical duplicate suffix, rename format changed to `YYYY.MM.DD Full Name_Vendor_Amount` (rgirish)
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Tow Foundation"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: Worker
|
|
5
|
+
client: tow-foundation
|
|
6
|
+
type: profile
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-06-11
|
|
9
|
+
owners: ["rgirish"]
|
|
10
|
+
files: []
|
|
11
|
+
related:
|
|
12
|
+
- clients/tow-foundation/features/receipt-processing.md
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Summary
|
|
16
|
+
|
|
17
|
+
Tow Foundation is a nonprofit client. Their integration automates credit card receipt
|
|
18
|
+
processing: receipts uploaded to SharePoint by cardholders are downloaded, extracted via
|
|
19
|
+
Talos AI, cross-verified against Amex/Mastercard billing statements, renamed, archived,
|
|
20
|
+
and exported as QuickBooks-ready Excel files.
|
|
21
|
+
|
|
22
|
+
No client database is used (no persistence beyond SharePoint and email).
|
|
23
|
+
|
|
24
|
+
## Contacts
|
|
25
|
+
|
|
26
|
+
- **Jheanelle Gordon** — primary contact; receives the processing completion email
|
|
27
|
+
- **Magdalena Minta** — CC on completion email
|
|
28
|
+
- devteam@togatech.com — BCC on all emails
|
|
29
|
+
|
|
30
|
+
## Key integration points
|
|
31
|
+
|
|
32
|
+
- Microsoft SharePoint (Microsoft Graph API) — receipt file storage
|
|
33
|
+
- Talos AI (`/api/ai/generate`) — structured receipt data extraction
|
|
34
|
+
- PhpSpreadsheet — Excel generation and statement parsing
|
|
35
|
+
- `_Email` — completion summary + fatal error notifications
|
package/knowledge/registry.json
CHANGED
|
@@ -5,5 +5,7 @@
|
|
|
5
5
|
{ "repo": "dbchanges2", "project": "Database Changes", "framework": "2.0", "role": "core", "dependsOn": [] },
|
|
6
6
|
{ "repo": "library", "project": "Library", "framework": "1.0", "role": "core", "dependsOn": [] },
|
|
7
7
|
{ "repo": "worker", "project": "Worker", "framework": "1.0", "role": "app", "dependsOn": [] },
|
|
8
|
-
{ "repo": "toga2-supply", "project": "TOGa Supply", "framework": "2.0", "role": "app", "dependsOn": ["api2"] }
|
|
8
|
+
{ "repo": "toga2-supply", "project": "TOGa Supply", "framework": "2.0", "role": "app", "dependsOn": ["api2"] },
|
|
9
|
+
{ "repo": "saml", "project": "SAML SSO Gateway", "framework": "2.0", "role": "app", "dependsOn": [] },
|
|
10
|
+
{ "repo": "toga2-view", "project": "TOGa View Frontend", "framework": "2.0", "role": "app", "dependsOn": ["api2"] }
|
|
9
11
|
]
|
package/package.json
CHANGED