@commandgarden/cli 2.14.1 → 2.14.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/node_modules/@commandgarden/app/dist/server/routes/skills.js +2 -2
- package/node_modules/@commandgarden/app/package.json +0 -3
- package/node_modules/@commandgarden/app/skills/cg/SKILL.md +85 -0
- package/node_modules/@commandgarden/app/skills/connector-authoring/RECON-PLAYBOOK.md +364 -0
- package/node_modules/@commandgarden/app/skills/connector-authoring/SKILL.md +160 -0
- package/node_modules/@commandgarden/chrome/package.json +0 -3
- package/node_modules/@commandgarden/daemon/package.json +0 -4
- package/node_modules/@commandgarden/shared/package.json +0 -3
- package/package.json +1 -1
|
@@ -4,10 +4,10 @@ import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
|
|
|
4
4
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
5
5
|
function findSkillsDir() {
|
|
6
6
|
const candidates = [
|
|
7
|
-
// From app/{dist,src}/server/routes/ → commandGarden/skills/
|
|
8
|
-
join(__dirname, '..', '..', '..', '..', 'skills'),
|
|
9
7
|
// From app/{dist,src}/server/routes/ → app/skills/ (bundled via prepare-bundle)
|
|
10
8
|
join(__dirname, '..', '..', '..', 'skills'),
|
|
9
|
+
// From app/{dist,src}/server/routes/ → monorepo root skills/ (local dev)
|
|
10
|
+
join(__dirname, '..', '..', '..', '..', '..', 'skills'),
|
|
11
11
|
];
|
|
12
12
|
for (const p of candidates) {
|
|
13
13
|
if (existsSync(p) && statSync(p).isDirectory())
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: cg
|
|
3
|
+
description: "Run commandGarden connectors via `cg` to query enterprise data from authenticated browser sessions. Use when the user mentions `cg`, commandGarden, connectors, or wants to fetch timetracking, room availability, security news, or data behind SSO."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
`cg` queries enterprise websites through **connectors** — declarative YAML pipelines executed via a Chrome extension using the user's live browser session. No credentials stored.
|
|
7
|
+
|
|
8
|
+
## Workflow
|
|
9
|
+
|
|
10
|
+
### 1. Verify readiness
|
|
11
|
+
|
|
12
|
+
Run `cg daemon status`. Proceed only when output shows "Daemon: running" and "Extension: connected". If not running, run `cg up`.
|
|
13
|
+
|
|
14
|
+
### 2. Discover
|
|
15
|
+
|
|
16
|
+
Run `cg list` to see all installed connectors. Each connector has a key in `site/name` format (e.g. `timetracking/report`). Filter large output: `cg list | Select-String "keyword"` (PowerShell) or `cg list | grep -i "keyword"` (bash).
|
|
17
|
+
|
|
18
|
+
### 3. Inspect
|
|
19
|
+
|
|
20
|
+
Run `cg inspect <site/name>` before running any connector. Shows required arguments, defaults, output columns, and pipeline steps. Never guess argument names.
|
|
21
|
+
|
|
22
|
+
### 4. Run
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
cg run <site/name> --format json [--arg value ...]
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
**Always `--format json`**. Output shape:
|
|
29
|
+
|
|
30
|
+
```json
|
|
31
|
+
{"ok": true, "connector": "site/name", "rowCount": N, "columns": [...], "data": [{...}, ...]}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### 5. Parse
|
|
35
|
+
|
|
36
|
+
Read the `data` array from JSON output. Column names match `columns`. `rowCount` confirms expected cardinality.
|
|
37
|
+
|
|
38
|
+
Errors return plain text starting with `Error:` — not JSON.
|
|
39
|
+
|
|
40
|
+
## Connectors
|
|
41
|
+
|
|
42
|
+
| Key | Description | Args |
|
|
43
|
+
|-----|-------------|------|
|
|
44
|
+
| `ado/git-commits` | ADO Git commits & PRs for a date range | `org`, `project`, `repo`, `fromDate`, `toDate`, `author` |
|
|
45
|
+
| `gcs/kb-pages` | GCS Knowledge Base page index | — |
|
|
46
|
+
| `gcs/kb-content` | Retrieve one GCS KB page as Markdown | `path` |
|
|
47
|
+
| `jira/my-tickets` | Jira tickets assigned to user | `fromDate`, `toDate`, `assignee` |
|
|
48
|
+
| `outlook/my-meetings` | User's Outlook Calendar for a day | `date` |
|
|
49
|
+
| `saba/pending-training` | Pending mandatory training from Saba | — |
|
|
50
|
+
| `socket/security-news` | Latest security news from Socket.dev | `tag` (optional) |
|
|
51
|
+
| `teams/room-availability` | Room free/busy for one room | `room`, `date` |
|
|
52
|
+
| `teams/rooms-availability` | Room free/busy for multiple rooms | `rooms`, `email`, `date` |
|
|
53
|
+
| `timetracking/projects` | Available projects & activities | `month` |
|
|
54
|
+
| `timetracking/report` | Monthly time-tracking report | `month` |
|
|
55
|
+
| `tldrsec/newsletter` | tl;dr sec newsletter issues | — |
|
|
56
|
+
| `tokenmaster/client-trustedby` | Clients that trust a given client | `clientid`, `region` |
|
|
57
|
+
| `tokenmaster/clients-list` | Clients managed by current user | `region` |
|
|
58
|
+
| `wiz/blog-security` | Wiz security blog posts | `tag` (optional) |
|
|
59
|
+
|
|
60
|
+
## Errors
|
|
61
|
+
|
|
62
|
+
| Output contains | Fix |
|
|
63
|
+
|---|---|
|
|
64
|
+
| "not running" / "No session token" | `cg up` |
|
|
65
|
+
| "not connected" | Open Chrome with the extension loaded |
|
|
66
|
+
| "not approved" | Add connector key to `security.approvedHighRisk` in `~/.commandgarden/config.yaml` |
|
|
67
|
+
| "back/forward cache" | Retry the command |
|
|
68
|
+
|
|
69
|
+
## Commands
|
|
70
|
+
|
|
71
|
+
| Command | What it does |
|
|
72
|
+
|---|---|
|
|
73
|
+
| `cg up` / `cg down` | Start / stop all services |
|
|
74
|
+
| `cg daemon status` | Health check (daemon + extension) |
|
|
75
|
+
| `cg list` | All installed connectors |
|
|
76
|
+
| `cg inspect <key>` | Connector args, columns, pipeline |
|
|
77
|
+
| `cg run <key> --format json` | Execute a connector |
|
|
78
|
+
| `cg validate <file>` | Validate a connector YAML file |
|
|
79
|
+
| `cg audit list --since <dur>` | Recent audit events (e.g. `7d`, `1h`) |
|
|
80
|
+
| `cg audit export --format json --since <dur>` | Export audit events as JSON |
|
|
81
|
+
| `cg config show` | View current configuration |
|
|
82
|
+
|
|
83
|
+
## Writing connectors
|
|
84
|
+
|
|
85
|
+
Use the [connector-authoring skill](../connector-authoring/SKILL.md) for the full recon→scaffold→implement→debug→polish sequence. Reference docs: [`connector-authoring.md`](../../docs/connector-authoring.md) and [`pipeline-reference.md`](../../docs/pipeline-reference.md). Place new connector YAML files in `~/.commandgarden/connectors/`. Validate with `cg validate <file>` before use.
|
|
@@ -0,0 +1,364 @@
|
|
|
1
|
+
# Recon Playbook
|
|
2
|
+
|
|
3
|
+
Ground-layer debugging for commandGarden connectors. Use when the audit log and basic probing (step 4 of SKILL.md) aren't enough to get data flowing.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Network interception hierarchy
|
|
8
|
+
|
|
9
|
+
From most visible (page JS) to most powerful (network stack):
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
window.fetch patch → Sees only page-initiated fetch() calls
|
|
13
|
+
↓ Invisible: SW-mediated, MCAS-wrapped, XHR
|
|
14
|
+
CDP Network.enable → Sees per-target requests (tab OR SW, not both)
|
|
15
|
+
↓ Invisible: may miss timing-dependent SW events
|
|
16
|
+
CDP Fetch.enable → Sees ALL requests at the network stack level
|
|
17
|
+
Sees: SW-mediated, MCAS-wrapped, proxied — everything
|
|
18
|
+
Caveat: response bodies may be base64-encoded
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Always start at the top. Escalate down only when the higher level captures nothing.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Technique 1: Strip the pipeline
|
|
26
|
+
|
|
27
|
+
Remove steps from the end until data flows. Isolate which step breaks.
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
# Keep only: navigate + wait
|
|
31
|
+
# Expected: ok: true, data: [], 0 rows — navigation works
|
|
32
|
+
cg run <site>/<name> --format json
|
|
33
|
+
|
|
34
|
+
# Add: fetch or extract
|
|
35
|
+
# Expected: ok: true, data: [...], N rows — data source works
|
|
36
|
+
cg run <site>/<name> --format json
|
|
37
|
+
|
|
38
|
+
# Add: map
|
|
39
|
+
# Check: field values aren't "undefined" — field names match
|
|
40
|
+
cg run <site>/<name> --format json
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
If `navigate + wait` fails, the problem is auth (SSO redirect, wrong URL, page blocked).
|
|
44
|
+
If `fetch` returns 0 rows, the problem is the API call (wrong URL, content type, wrapper object).
|
|
45
|
+
If `map` produces `"undefined"`, the problem is field name casing.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Technique 2: Inspect raw API responses
|
|
50
|
+
|
|
51
|
+
For `fetch` connectors — remove the `map` step and examine raw output:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
cg run <site>/<name> --format json
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**0 rows, no error:**
|
|
58
|
+
- API returned non-JSON → add `headers: { Accept: "application/json" }` to the `fetch` step.
|
|
59
|
+
- API returned a wrapper object like `{ "items": [...] }` → add `dataPath: "items"`.
|
|
60
|
+
- `fetch` ran on the wrong tab or URL → check the audit log `steps[]` for the fetch step's error/timing.
|
|
61
|
+
|
|
62
|
+
**Data present but wrong shape:**
|
|
63
|
+
- Field names are case-sensitive. Compare the raw JSON keys with your `map` expressions exactly.
|
|
64
|
+
- Arrays in responses (e.g. `["ALICE", "BOB"]`) auto-join to `"ALICE,BOB"` in map expressions.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## Technique 3: Console logging in js_evaluate
|
|
69
|
+
|
|
70
|
+
Add `console.log()` to the eval.js file. Output appears in the **target tab's** DevTools console (not the extension's service worker console).
|
|
71
|
+
|
|
72
|
+
```js
|
|
73
|
+
console.log('[cg] token:', token ? 'found (' + token.slice(0, 8) + '...)' : 'MISSING');
|
|
74
|
+
console.log('[cg] API status:', resp.status);
|
|
75
|
+
console.log('[cg] response preview:', JSON.stringify(data).slice(0, 300));
|
|
76
|
+
console.log('[cg] row count:', rows.length);
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
After the debug session, remove all `[cg]` logs — they're visible to anyone with DevTools open.
|
|
80
|
+
|
|
81
|
+
**DevTools not showing logs?** The eval.js runs in the page's MAIN world. Make sure you're looking at the correct tab's console, not the extension's background page.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Technique 4: window.fetch interception
|
|
86
|
+
|
|
87
|
+
Capture data from requests the page makes on its own — not your `fetch` step, but the page's own API calls triggered by UI interactions.
|
|
88
|
+
|
|
89
|
+
```js
|
|
90
|
+
// Install the interceptor BEFORE triggering the UI action
|
|
91
|
+
if (!window.__rfb) {
|
|
92
|
+
window.__rfb = [];
|
|
93
|
+
const origFetch = window.fetch;
|
|
94
|
+
window.fetch = function() {
|
|
95
|
+
const args = arguments;
|
|
96
|
+
const url = (args[0]?.url || args[0] || '').toString();
|
|
97
|
+
const body = (args[1]?.body || '').toString();
|
|
98
|
+
return origFetch.apply(this, args).then(r => {
|
|
99
|
+
if (url.includes('YOUR_KEYWORD')) {
|
|
100
|
+
r.clone().text().then(t => {
|
|
101
|
+
window.__rfb.push({ url, reqBody: body, resBody: t });
|
|
102
|
+
}).catch(() => {});
|
|
103
|
+
}
|
|
104
|
+
return r;
|
|
105
|
+
});
|
|
106
|
+
};
|
|
107
|
+
}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Then trigger the UI action (click a button, type in a field), wait, and read:
|
|
111
|
+
|
|
112
|
+
```js
|
|
113
|
+
await new Promise(r => setTimeout(r, 5000));
|
|
114
|
+
const captured = window.__rfb;
|
|
115
|
+
window.__rfb = [];
|
|
116
|
+
if (!captured.length) throw new Error('No requests captured — check URL keyword');
|
|
117
|
+
|
|
118
|
+
// Parse and process
|
|
119
|
+
const data = JSON.parse(captured[0].resBody);
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Gotcha: timing.** The interceptor must be installed before the page makes the request. If the page fires requests on load, install the interceptor, then re-trigger (e.g., navigate, click a refresh button).
|
|
123
|
+
|
|
124
|
+
**Gotcha: cloning.** Always use `r.clone().text()` — reading the original response body consumes it and breaks the page.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Technique 5: CDP Fetch.enable
|
|
129
|
+
|
|
130
|
+
When `window.fetch` interception captures nothing — the request uses a transport invisible to page-level JS.
|
|
131
|
+
|
|
132
|
+
**Common causes:**
|
|
133
|
+
- **MCAS proxy** (`*.mcas.ms`) — Microsoft Defender for Cloud Apps injects `js-wrapper.js` that handles requests outside `window.fetch`
|
|
134
|
+
- **Service Worker** — SW intercepts and replays requests on a separate CDP target
|
|
135
|
+
- **Custom XHR wrappers** — legacy apps using `XMLHttpRequest` with custom transport layers
|
|
136
|
+
|
|
137
|
+
### How to verify the gap
|
|
138
|
+
|
|
139
|
+
Widen the interceptor to log ALL fetch calls, not just your keyword:
|
|
140
|
+
|
|
141
|
+
```js
|
|
142
|
+
window.fetch = function() {
|
|
143
|
+
const url = (arguments[0]?.url || arguments[0] || '').toString();
|
|
144
|
+
console.log('[cg fetch]', url.slice(0, 120));
|
|
145
|
+
return origFetch.apply(this, arguments);
|
|
146
|
+
};
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
If the request you're looking for doesn't appear in the console but the data renders in the UI → it's using a transport below `window.fetch`.
|
|
150
|
+
|
|
151
|
+
### The fix: CDP Fetch domain
|
|
152
|
+
|
|
153
|
+
Requires modifying `chrome/src/background/chrome-adapter.ts`. The `Fetch` CDP domain intercepts at the browser's network stack — below the SW and MCAS layers.
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
// In attachDebugger():
|
|
157
|
+
await chrome.debugger.sendCommand({ tabId }, 'Fetch.enable', {
|
|
158
|
+
patterns: [{ urlPattern: '*your-pattern*', requestStage: 'Response' }],
|
|
159
|
+
});
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
Handle `Fetch.requestPaused` events:
|
|
163
|
+
|
|
164
|
+
```typescript
|
|
165
|
+
// 1. Read the response body
|
|
166
|
+
const result = await chrome.debugger.sendCommand(
|
|
167
|
+
debuggee, 'Fetch.getResponseBody', { requestId }
|
|
168
|
+
);
|
|
169
|
+
const raw = result as { body: string; base64Encoded: boolean };
|
|
170
|
+
|
|
171
|
+
// 2. Decode if base64 (MCAS proxied responses are always base64)
|
|
172
|
+
const body = raw.base64Encoded ? atob(raw.body) : (raw.body || '');
|
|
173
|
+
|
|
174
|
+
// 3. Check for your signal
|
|
175
|
+
if (body.includes('yourKeyword')) {
|
|
176
|
+
// Inject into page context via chrome.scripting.executeScript
|
|
177
|
+
await chrome.scripting.executeScript({
|
|
178
|
+
target: { tabId },
|
|
179
|
+
world: 'MAIN',
|
|
180
|
+
func: (data) => { window.__rfb.push(data); },
|
|
181
|
+
args: [{ body }],
|
|
182
|
+
});
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
// 4. ALWAYS continue the request or the page hangs
|
|
186
|
+
await chrome.debugger.sendCommand(
|
|
187
|
+
debuggee, 'Fetch.continueRequest', { requestId }
|
|
188
|
+
).catch(() => {}); // swallow errors on detached targets
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
### Critical gotchas
|
|
192
|
+
|
|
193
|
+
1. **base64 decoding** — `Fetch.getResponseBody` returns base64 for MCAS-proxied responses. Check the `base64Encoded` flag. Always `atob()` before string matching.
|
|
194
|
+
|
|
195
|
+
2. **Must continue** — every `Fetch.requestPaused` must be followed by `Fetch.continueRequest`, `Fetch.fulfillRequest`, or `Fetch.failRequest`. Missing this hangs the page permanently. Use `.catch()` to swallow errors from detached targets.
|
|
196
|
+
|
|
197
|
+
3. **TypeScript** — `atob` is available in the Chrome extension service worker at runtime but not typed. Add at file top:
|
|
198
|
+
```typescript
|
|
199
|
+
declare function atob(data: string): string;
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
4. **Service Worker targets** — SW network events are on a separate CDP target. Attach to both the tab and the SW:
|
|
203
|
+
```typescript
|
|
204
|
+
const targets = await chrome.debugger.getTargets();
|
|
205
|
+
const sw = targets.find(t =>
|
|
206
|
+
t.type === 'service_worker' && t.url.startsWith(origin)
|
|
207
|
+
);
|
|
208
|
+
if (sw?.id) {
|
|
209
|
+
await chrome.debugger.attach({ targetId: sw.id }, '1.3');
|
|
210
|
+
await chrome.debugger.sendCommand(
|
|
211
|
+
{ targetId: sw.id }, 'Fetch.enable',
|
|
212
|
+
{ patterns: [{ urlPattern: '*pattern*', requestStage: 'Response' }] }
|
|
213
|
+
);
|
|
214
|
+
}
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
5. **Chrome `debugger` permission** — the extension manifest needs:
|
|
218
|
+
```json
|
|
219
|
+
{ "permissions": ["debugger"] }
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
Full investigation writeup: `docs/teams-room-availability-fix.md`.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## Technique 6: Auth patterns
|
|
227
|
+
|
|
228
|
+
### SSO/MFA redirect polling
|
|
229
|
+
|
|
230
|
+
The first `navigate` triggers SSO. Build in wait time:
|
|
231
|
+
|
|
232
|
+
```js
|
|
233
|
+
// Poll for a signal that auth completed
|
|
234
|
+
const deadline = Date.now() + 30000;
|
|
235
|
+
while (Date.now() < deadline) {
|
|
236
|
+
// Signal: a DOM element that only renders when logged in
|
|
237
|
+
if (document.querySelector('.user-avatar')) break;
|
|
238
|
+
// Or: check if we're still on a login page
|
|
239
|
+
if (/login|oauth|signin|sso/i.test(location.href)) {
|
|
240
|
+
await new Promise(r => setTimeout(r, 2000));
|
|
241
|
+
continue;
|
|
242
|
+
}
|
|
243
|
+
break;
|
|
244
|
+
}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
### MSAL token extraction (Azure AD apps)
|
|
248
|
+
|
|
249
|
+
Many Microsoft/enterprise apps store tokens in sessionStorage. See `connectors/lib/msal-token.js` for the canonical pattern:
|
|
250
|
+
|
|
251
|
+
```js
|
|
252
|
+
const deadline = Date.now() + 60000;
|
|
253
|
+
let token;
|
|
254
|
+
while (Date.now() < deadline) {
|
|
255
|
+
const key = Object.keys(sessionStorage).find(k => k.includes('accesstoken'));
|
|
256
|
+
if (key) {
|
|
257
|
+
try {
|
|
258
|
+
const data = JSON.parse(sessionStorage.getItem(key));
|
|
259
|
+
// Check token isn't about to expire (30s buffer)
|
|
260
|
+
if (Number(data.expiresOn) > Math.floor(Date.now() / 1000) + 30) {
|
|
261
|
+
token = data.secret;
|
|
262
|
+
break;
|
|
263
|
+
}
|
|
264
|
+
} catch (_) {}
|
|
265
|
+
}
|
|
266
|
+
await new Promise(r => setTimeout(r, 1000));
|
|
267
|
+
}
|
|
268
|
+
if (!token) throw new Error('No valid MSAL access token — log in and retry');
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Use the token as a Bearer header:
|
|
272
|
+
```js
|
|
273
|
+
const resp = await fetch(apiUrl, {
|
|
274
|
+
headers: { Authorization: 'Bearer ' + token, Accept: 'application/json' }
|
|
275
|
+
});
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
### Session cookies (fetch pattern)
|
|
279
|
+
|
|
280
|
+
For `fetch` pattern connectors, session cookies are sent automatically — `fetchFromPage` uses `credentials: 'include'`. Just `navigate` to the domain first to establish the session through SSO.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
## Technique 7: SPA rendering delays
|
|
285
|
+
|
|
286
|
+
SPAs (React, Angular, Vue) render data after the initial page load. The `wait` step checks that a selector *exists* in the DOM — not that data has populated.
|
|
287
|
+
|
|
288
|
+
### In eval.js — poll for content
|
|
289
|
+
|
|
290
|
+
```js
|
|
291
|
+
const deadline = Date.now() + 30000;
|
|
292
|
+
while (Date.now() < deadline) {
|
|
293
|
+
const rows = document.querySelectorAll('.data-table tbody tr');
|
|
294
|
+
if (rows.length > 0) break;
|
|
295
|
+
await new Promise(r => setTimeout(r, 1000));
|
|
296
|
+
}
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
### In YAML — chain wait + extract
|
|
300
|
+
|
|
301
|
+
```yaml
|
|
302
|
+
- step: wait
|
|
303
|
+
selector: ".data-table tbody tr" # wait for actual rows, not just the table
|
|
304
|
+
timeout: 30000
|
|
305
|
+
- step: extract
|
|
306
|
+
selector: ".data-table tbody tr"
|
|
307
|
+
fields:
|
|
308
|
+
name: "td:nth-child(1)"
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
---
|
|
312
|
+
|
|
313
|
+
## Technique 8: Driving UI interactions in eval.js
|
|
314
|
+
|
|
315
|
+
When you need to click buttons, type in fields, or navigate within an SPA — all from eval.js. See `connectors/teams-room-availability.eval.js` for a production example of all three.
|
|
316
|
+
|
|
317
|
+
### Clicking by accessible name
|
|
318
|
+
|
|
319
|
+
```js
|
|
320
|
+
function clickByName(name) {
|
|
321
|
+
const els = Array.from(document.querySelectorAll('button, a, [role=button]'));
|
|
322
|
+
const el = els.find(e => {
|
|
323
|
+
if (!e.offsetParent) return false; // skip hidden
|
|
324
|
+
const label = (e.getAttribute('aria-label') || '').trim();
|
|
325
|
+
const text = (e.textContent || '').replace(/\s+/g, ' ').trim();
|
|
326
|
+
return label === name || label.includes(name) || text.includes(name);
|
|
327
|
+
});
|
|
328
|
+
if (!el) return false;
|
|
329
|
+
el.scrollIntoView({ block: 'center' });
|
|
330
|
+
el.click();
|
|
331
|
+
return true;
|
|
332
|
+
}
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
### Typing into React-controlled inputs
|
|
336
|
+
|
|
337
|
+
React overrides the native `value` setter. Use the prototype setter to bypass:
|
|
338
|
+
|
|
339
|
+
```js
|
|
340
|
+
function typeText(selector, text) {
|
|
341
|
+
const el = document.querySelector(selector);
|
|
342
|
+
if (!el) throw new Error(`Element "${selector}" not found`);
|
|
343
|
+
el.focus();
|
|
344
|
+
const setter = Object.getOwnPropertyDescriptor(
|
|
345
|
+
window.HTMLInputElement.prototype, 'value'
|
|
346
|
+
).set;
|
|
347
|
+
setter.call(el, text);
|
|
348
|
+
el.dispatchEvent(new Event('input', { bubbles: true }));
|
|
349
|
+
el.dispatchEvent(new Event('change', { bubbles: true }));
|
|
350
|
+
}
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
### Waiting for a UI reaction
|
|
354
|
+
|
|
355
|
+
After clicking or typing, poll for the expected outcome — never use fixed sleeps alone:
|
|
356
|
+
|
|
357
|
+
```js
|
|
358
|
+
clickByName('Submit');
|
|
359
|
+
const deadline = Date.now() + 10000;
|
|
360
|
+
while (Date.now() < deadline) {
|
|
361
|
+
if (document.querySelector('.results-loaded')) break;
|
|
362
|
+
await new Promise(r => setTimeout(r, 500));
|
|
363
|
+
}
|
|
364
|
+
```
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: connector-authoring
|
|
3
|
+
description: "Connector authoring for commandGarden. Use when creating a new connector YAML, adding a data source, or debugging why a connector returns 0 rows."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Build a new commandGarden connector from recon through polish. Each step has a checkable completion criterion.
|
|
7
|
+
|
|
8
|
+
## Step 1: Recon the target site
|
|
9
|
+
|
|
10
|
+
Open the target site in Chrome with DevTools. Goal: find where the data lives and which extraction pattern fits.
|
|
11
|
+
|
|
12
|
+
**Check in this order (cheapest capability first):**
|
|
13
|
+
|
|
14
|
+
1. **Network tab → XHR/Fetch** — filter requests while using the UI. JSON API responses → **`fetch → map`** (capabilities: `navigate`, `cookie_read`). Safest pattern — prefer it.
|
|
15
|
+
|
|
16
|
+
2. **Elements tab → DOM** — repeating elements (table rows, cards) → **`extract`** (`navigate`, `dom_read`). Nested tree (sidebar nav) → **`extract_tree`** (add `dom_write` if collapsed sections need `click_all`).
|
|
17
|
+
|
|
18
|
+
3. **Network tab → page-initiated requests** — data from requests the page makes on its own (GraphQL, polling) → **`intercept`** step with `urlPattern` (`navigate`, `intercept_response`). Middle ground between `fetch` and `js_evaluate`.
|
|
19
|
+
|
|
20
|
+
4. **Source tab → `<script id="__NEXT_DATA__">`** or `__remixContext` — framework-embedded page data → **`js_evaluate`** parsing the script tag.
|
|
21
|
+
|
|
22
|
+
5. **Console → `sessionStorage`** — MSAL/OAuth tokens needed as Bearer for API calls → **`js_evaluate`** polling sessionStorage then calling fetch with the token.
|
|
23
|
+
|
|
24
|
+
6. **Network tab → invisible transports** — requests missing from `window.fetch` interception (MCAS proxy, Service Workers) → **`js_evaluate`** with fetch interception, potentially requiring CDP `Fetch.enable`. Read [`RECON-PLAYBOOK.md`](RECON-PLAYBOOK.md) technique 5 before going this route.
|
|
25
|
+
|
|
26
|
+
**Record during recon:**
|
|
27
|
+
- Exact hostname(s) → `domains`
|
|
28
|
+
- API endpoint or DOM selectors
|
|
29
|
+
- Response shape (field names, types) → `columns`
|
|
30
|
+
- Auth mechanism (SSO redirect, MFA, token polling, session cookies)
|
|
31
|
+
- Proxy layers (MCAS `*.mcas.ms`, Cloudflare, etc.)
|
|
32
|
+
|
|
33
|
+
**Completion:** You can name the pattern and have the target URL, data shape, and auth mechanism documented.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Step 2: Scaffold the YAML
|
|
38
|
+
|
|
39
|
+
Create the `.yaml` file in `~/.commandgarden/connectors/` for rapid iteration (daemon loads this directory without rebuild).
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
site: <sitename>
|
|
43
|
+
name: <command-name>
|
|
44
|
+
version: "1.0"
|
|
45
|
+
description: "<what data this connector returns>"
|
|
46
|
+
access: read
|
|
47
|
+
|
|
48
|
+
domains:
|
|
49
|
+
- "<exact hostname from recon>"
|
|
50
|
+
capabilities:
|
|
51
|
+
- navigate
|
|
52
|
+
# add others based on chosen pattern
|
|
53
|
+
|
|
54
|
+
args:
|
|
55
|
+
- name: <argname>
|
|
56
|
+
type: string
|
|
57
|
+
required: true
|
|
58
|
+
help: "<what this arg does>"
|
|
59
|
+
# optional: pattern, enum, default
|
|
60
|
+
|
|
61
|
+
columns:
|
|
62
|
+
- name: <fieldname>
|
|
63
|
+
type: string # or number, boolean
|
|
64
|
+
|
|
65
|
+
pipeline: []
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
The full connector schema is defined in `shared/src/connector.ts` (Zod). Validation rules:
|
|
69
|
+
- Every hostname in `navigate`/`fetch`/`cookie` URLs must appear in `domains`.
|
|
70
|
+
- Every pipeline step's required capability must appear in `capabilities`.
|
|
71
|
+
- `js_evaluate` steps require either `code` or `file`.
|
|
72
|
+
|
|
73
|
+
Run `cg validate <path>` — it should report only the empty-pipeline error, nothing else.
|
|
74
|
+
|
|
75
|
+
**Completion:** `cg validate` reports only the pipeline-length schema error.
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## Step 3: Implement the pipeline
|
|
80
|
+
|
|
81
|
+
Fill in `pipeline` based on the pattern from recon.
|
|
82
|
+
|
|
83
|
+
Read [`docs/connector-authoring.md`](../../docs/connector-authoring.md) for pattern templates (fetch→map, extract, extract_tree, js_evaluate, __NEXT_DATA__) and [`docs/pipeline-reference.md`](../../docs/pipeline-reference.md) for the full step reference.
|
|
84
|
+
|
|
85
|
+
**Key rules for `js_evaluate` eval.js files:**
|
|
86
|
+
|
|
87
|
+
- **No IIFE wrapper.** The runner wraps code in `new AsyncFunction()`. A `return` inside `(function(){ ... })()` exits the inner function — the outer function gets `undefined` and the pipeline silently produces 0 rows.
|
|
88
|
+
- **`await` works** at the top level.
|
|
89
|
+
- **Template variables** `${{ args.name }}` are interpolated before execution. Quote them: `const x = '${{ args.date }}'`.
|
|
90
|
+
- **Return** an array of objects to set pipeline data. Or use `as:` on the step to store a single value as a variable.
|
|
91
|
+
- **Poll for auth** — SSO redirects take time. Poll `sessionStorage` or a DOM signal in a loop with a 30–60s deadline.
|
|
92
|
+
|
|
93
|
+
Create the `.eval.js` file **next to** the YAML. See existing examples in `connectors/` for each pattern.
|
|
94
|
+
|
|
95
|
+
**Completion:** `cg validate <path>` passes clean. If using `js_evaluate`, the `.eval.js` file exists.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Step 4: Debug loop
|
|
100
|
+
|
|
101
|
+
The loop: **run → read audit → probe → fix → repeat.**
|
|
102
|
+
|
|
103
|
+
### First run
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
# Approve if using js_evaluate
|
|
107
|
+
cg config set security.approvedHighRisk <site>/<name>
|
|
108
|
+
|
|
109
|
+
cg run <site>/<name> --format json
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### Read the audit log (always do this first)
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
cg audit export --format json --since 5m
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Check `steps[]` — per-step timing and errors. See the symptom→cause table in [`docs/connector-authoring.md` § Debugging Connectors](../../docs/connector-authoring.md#debugging-connectors).
|
|
119
|
+
|
|
120
|
+
### When the audit log isn't enough
|
|
121
|
+
|
|
122
|
+
Read [`RECON-PLAYBOOK.md`](RECON-PLAYBOOK.md) for ground-layer probing:
|
|
123
|
+
|
|
124
|
+
- **Strip the pipeline** — remove steps from the end until data flows, add back one at a time.
|
|
125
|
+
- **Raw output** — remove `map` step, run with `--format json` to see actual field names.
|
|
126
|
+
- **Console injection** — `console.log()` in eval.js, watch in DevTools.
|
|
127
|
+
- **window.fetch interception** — capture requests the page makes on its own.
|
|
128
|
+
- **CDP Fetch.enable** — when `window.fetch` misses requests (MCAS, Service Workers).
|
|
129
|
+
|
|
130
|
+
### Reload cycle
|
|
131
|
+
|
|
132
|
+
The daemon caches eval.js contents at startup via `ConnectorRegistry.resolveFileRefs()`.
|
|
133
|
+
|
|
134
|
+
| What changed | Minimum reload |
|
|
135
|
+
|---|---|
|
|
136
|
+
| WIP YAML/eval.js in `~/.commandgarden/connectors/` | Restart daemon: `cg down && cg up` |
|
|
137
|
+
| Source YAML only (in `connectors/`) | `npm run build -w daemon && cg down && cg up` |
|
|
138
|
+
| Source YAML + eval.js | `npm run build -w daemon && cg down && cg up` |
|
|
139
|
+
| Extension code (chrome-adapter.ts) | `npm run build && cg down && cg up` + reload at `chrome://extensions` |
|
|
140
|
+
|
|
141
|
+
**Fastest path:** Put WIP connector in `~/.commandgarden/connectors/`. Edit → restart daemon → test. No build step.
|
|
142
|
+
|
|
143
|
+
**Completion:** `cg run` returns expected data rows consistently across 3+ runs. Audit log shows no unexpected step errors or delays.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## Step 5: Polish
|
|
148
|
+
|
|
149
|
+
1. **Args** — add `pattern` for format validation (`"^\\d{4}-\\d{2}$"`), `enum` for fixed choices, `default` for optional args.
|
|
150
|
+
2. **Columns** — declare every output field with correct `type`. Drives CLI table formatting.
|
|
151
|
+
3. **Error messages** — in eval.js, throw with actionable text: `'Not signed in — log in and retry'`, `'No results for "${query}"'`.
|
|
152
|
+
4. **Description** — one line: what data the connector returns, not how.
|
|
153
|
+
5. **Validate + inspect:**
|
|
154
|
+
```bash
|
|
155
|
+
cg validate <path>
|
|
156
|
+
cg list # shows in connector list
|
|
157
|
+
cg inspect <site>/<name> # shows full definition with args
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
**Completion:** `cg validate` passes. `cg list` shows the connector. `cg inspect` shows args with help text. Bad args produce a helpful error. `cg run` with `--format table` formats columns correctly.
|