explorbot 0.4.2 → 0.4.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/explorbot-cli.ts +6 -1
- package/boat/api-tester/src/apibot.ts +8 -13
- package/boat/api-tester/src/cli.ts +7 -3
- package/boat/api-tester/src/config.ts +45 -9
- package/boat/prima/src/cli.ts +33 -99
- package/boat/prima/src/envelope.ts +3 -1
- package/boat/prima/src/help.ts +72 -0
- package/boat/prima/src/prima.ts +33 -43
- package/dist/bin/explorbot-cli.js +5 -1
- package/dist/boat/api-tester/src/apibot.js +7 -6
- package/dist/boat/api-tester/src/cli.js +9 -3
- package/dist/boat/api-tester/src/config.js +32 -6
- package/dist/boat/prima/src/cli.js +30 -86
- package/dist/boat/prima/src/envelope.js +2 -1
- package/dist/boat/prima/src/help.js +63 -0
- package/dist/boat/prima/src/prima.js +29 -41
- package/dist/package.json +1 -1
- package/dist/src/action-result.d.ts +3 -0
- package/dist/src/action-result.js +5 -0
- package/dist/src/action.js +12 -1
- package/dist/src/ai/fisherman/request-haul.d.ts +1 -0
- package/dist/src/ai/fisherman/request-haul.js +3 -0
- package/dist/src/ai/fisherman/tools.d.ts +50 -0
- package/dist/src/ai/{fisherman-tools.js → fisherman/tools.js} +78 -13
- package/dist/src/ai/fisherman.d.ts +12 -3
- package/dist/src/ai/fisherman.js +89 -13
- package/dist/src/ai/pilot.d.ts +13 -1
- package/dist/src/ai/pilot.js +20 -7
- package/dist/src/ai/researcher/deep-analysis.d.ts +1 -1
- package/dist/src/ai/researcher/deep-analysis.js +4 -1
- package/dist/src/ai/researcher/sections.d.ts +1 -1
- package/dist/src/ai/researcher/sections.js +2 -1
- package/dist/src/ai/researcher.js +25 -11
- package/dist/src/ai/rules.js +2 -0
- package/dist/src/ai/tester.d.ts +1 -0
- package/dist/src/ai/tester.js +27 -33
- package/dist/src/ai/tools.js +5 -0
- package/dist/src/api/request-result.js +3 -1
- package/dist/src/api/request-store.d.ts +6 -1
- package/dist/src/api/request-store.js +55 -17
- package/dist/src/api/xhr-capture.d.ts +2 -0
- package/dist/src/api/xhr-capture.js +35 -10
- package/dist/src/commands/config-command.js +6 -2
- package/dist/src/commands/help-json-command.d.ts +31 -0
- package/dist/src/commands/help-json-command.js +58 -0
- package/dist/src/config.d.ts +3 -0
- package/dist/src/config.js +14 -0
- package/dist/src/state-manager.js +5 -1
- package/docs/api-testing/basics.md +12 -4
- package/docs/reference/commands.md +2 -0
- package/docs/reference/configuration.md +4 -0
- package/docs/superpowers/plans/2026-09-03-fisherman-query-api.md +1361 -0
- package/docs/workflow/agentic-usage.md +15 -1
- package/package.json +1 -1
- package/src/action-result.ts +7 -0
- package/src/action.ts +14 -2
- package/src/ai/fisherman/request-haul.ts +4 -0
- package/src/ai/{fisherman-tools.ts → fisherman/tools.ts} +93 -20
- package/src/ai/fisherman.ts +104 -15
- package/src/ai/pilot.ts +20 -7
- package/src/ai/researcher/deep-analysis.ts +4 -2
- package/src/ai/researcher/sections.ts +2 -2
- package/src/ai/researcher.ts +28 -11
- package/src/ai/rules.ts +2 -0
- package/src/ai/tester.ts +25 -30
- package/src/ai/tools.ts +6 -0
- package/src/api/request-result.ts +2 -1
- package/src/api/request-store.ts +58 -18
- package/src/api/xhr-capture.ts +39 -11
- package/src/commands/config-command.ts +4 -1
- package/src/commands/help-json-command.ts +74 -0
- package/src/config.ts +16 -0
- package/src/state-manager.ts +6 -1
- package/dist/src/ai/fisherman-tools.d.ts +0 -147
|
@@ -0,0 +1,1361 @@
|
|
|
1
|
+
# Fisherman queryApi Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** Pilot can ask Fisherman what data already exists — at any point in a test run, not only as a precondition at the start — through a read-only `queryApi(question)` tool that answers in prose from real GET responses.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Four layers change. `XhrCapture` starts recording successful GET requests as endpoint-only entries (path + query params, no response body), so replicate mode knows read endpoints at all. `RequestStore` deduplicates those structurally and gains a read/write family parameter on its scoping and endpoint-list methods. `createFishermanTools` gains a `readOnly` flag that narrows the `request` method enum to `GET`, returns a truncated response body for the model to quote, and swaps `finish`'s schema to `{ answer }`. `Fisherman` gains `lookupData()` next to `prepareData()`, both driving one extracted `runSession()` loop. `Pilot` exposes `queryApi` beside `precondition` from the same builder, which `sendToPilot` already spreads into planning, new-page review and progress analysis.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Bun, TypeScript, Zod (tool schemas), Vercel AI SDK `tool()`, bun:test, `@copilotkit/aimock` for agent integration tests.
|
|
10
|
+
|
|
11
|
+
**Spec:** None — design was settled in the brainstorming session of 2026-09-03 and is captured in the Design Decisions section below.
|
|
12
|
+
|
|
13
|
+
## Global Constraints
|
|
14
|
+
|
|
15
|
+
- Bun only — never Node.js. Run tests with `bun test`.
|
|
16
|
+
- Work in the worktree `/home/davert/projects/explorbot-fisherman-fetch`, branch `fisherman-fetch` (branched from `main`).
|
|
17
|
+
- No code comments unless explicitly requested.
|
|
18
|
+
- No ternary operators. No `...(cond ? {k: v} : {})` spread. Premature exit over if/else.
|
|
19
|
+
- Private methods go after public methods. Types go at the end of the file.
|
|
20
|
+
- Prompts must be general — never encode an example from a debug session or a specific site into a prompt, rule or tool description.
|
|
21
|
+
- Never hardcode locators, site names, or field names into source.
|
|
22
|
+
- Run `bun run format` after each code change, before each commit.
|
|
23
|
+
- Run `bun test tests/integration/` before any commit that touches a prompt, rule, tool schema or system message.
|
|
24
|
+
- NEVER trigger the regression CI workflow (no `regression` label, no `gh workflow run regression.yml`). Only the user does that.
|
|
25
|
+
|
|
26
|
+
## Design Decisions
|
|
27
|
+
|
|
28
|
+
Settled with the user before this plan was written. An implementer who disagrees should raise it, not silently deviate.
|
|
29
|
+
|
|
30
|
+
1. **GET capture is endpoint-only.** Successful (`200`) GET XHRs are recorded with their path and query string but **no response body**. The model figures out shapes from live responses at lookup time; storing bodies for every list fetch on every page load would flood `output/requests/`.
|
|
31
|
+
2. **The answer is prose, not a structured item list.** `lookupData` returns the model's own sentence quoting concrete names and ids. This is why the read-mode `request` tool must return a truncated raw body — without it the model has nothing real to quote.
|
|
32
|
+
3. **Read-only is a schema guarantee, not a prompt rule.** In `readOnly` mode the `request` tool's `method` enum contains only `GET`. A prompt instruction would be a soft constraint; a Zod enum is a hard one.
|
|
33
|
+
4. **`prepareData` keeps seeing GET endpoints in achieve mode.** Only the *read* family filters the OpenAPI list down to `GET`. Write preparation legitimately needs a GET to find a parent id, and today's behaviour must not regress.
|
|
34
|
+
5. **The tool is called `queryApi`.** `precondition` creates, `queryApi` reads. Naming the mechanism keeps the model from confusing it with the page-facing `verify` tool.
|
|
35
|
+
6. **A lookup produces a note, never a step.** Steps become generated CodeceptJS; an API read is not a UI action and must not leak into a generated test.
|
|
36
|
+
|
|
37
|
+
Out of scope: giving Tester the tool, caching repeated lookups within a test, and replacing the vision-based `checkDataAvailability` fallback with a lookup.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
### Task 1: Record successful GET endpoints
|
|
42
|
+
|
|
43
|
+
**Files:**
|
|
44
|
+
- Modify: `src/api/request-result.ts:126` (the `writeFileSync` of the response file at the end of `save`)
|
|
45
|
+
- Modify: `src/api/request-store.ts:8-18` (new field), `src/api/request-store.ts:20-23` (near `addCapturedRequest`), `src/api/request-store.ts:143-159` (`loadFromDisk`), file bottom (new helper)
|
|
46
|
+
- Modify: `src/api/xhr-capture.ts:33-63` (`captureResponse`), plus a new private method after it
|
|
47
|
+
- Test: `tests/unit/request-store.test.ts`, and create `tests/unit/xhr-capture.test.ts`
|
|
48
|
+
|
|
49
|
+
**Interfaces:**
|
|
50
|
+
- Consumes: existing `RequestResult` (`method`, `path`, `fullUrl`, `requestHeaders`, `status`, `isWrite`, `rawResponseBodyValue` setter, `save(outputDir)`), `generateRequestId(method, path, prefix)`, the module-private `normalizePathPattern(urlPath)` already at the bottom of `request-store.ts`.
|
|
51
|
+
- Produces: `RequestStore.addReadRequest(result: RequestResult): void` — stores a non-write capture only when its `METHOD + normalized path + sorted query-param names` key has not been seen, then persists it. Task 2 relies on read captures living in the same `capturedRequests` array as writes, distinguished only by `isWrite`.
|
|
52
|
+
|
|
53
|
+
- [ ] **Step 1: Write the failing tests**
|
|
54
|
+
|
|
55
|
+
Add this describe block at the end of `tests/unit/request-store.test.ts`. It reuses the file's existing `makeRequest` helper and adds a GET-specific one, because `makeRequest` sets `fullUrl` to the path with no query string.
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
describe('read endpoint capture', () => {
|
|
59
|
+
let outputDir: string;
|
|
60
|
+
|
|
61
|
+
beforeEach(() => {
|
|
62
|
+
outputDir = mkdtempSync(join(tmpdir(), 'reqstore-read-'));
|
|
63
|
+
});
|
|
64
|
+
|
|
65
|
+
afterEach(() => {
|
|
66
|
+
if (existsSync(outputDir)) rmSync(outputDir, { recursive: true, force: true });
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
function makeGet(urlPath: string, search = '', id?: string): RequestResult {
|
|
70
|
+
return new RequestResult({
|
|
71
|
+
id: id || `get_${urlPath}${search}`,
|
|
72
|
+
method: 'GET',
|
|
73
|
+
path: urlPath,
|
|
74
|
+
fullUrl: `${urlPath}${search}`,
|
|
75
|
+
requestHeaders: {},
|
|
76
|
+
status: 200,
|
|
77
|
+
statusText: '200',
|
|
78
|
+
responseHeaders: {},
|
|
79
|
+
timing: 0,
|
|
80
|
+
timestamp: new Date(),
|
|
81
|
+
});
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
it('keeps one entry when the same read endpoint is fetched repeatedly', () => {
|
|
85
|
+
const store = new RequestStore(outputDir);
|
|
86
|
+
|
|
87
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g1'));
|
|
88
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g2'));
|
|
89
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g3'));
|
|
90
|
+
|
|
91
|
+
expect(store.getCapturedRequests()).toHaveLength(1);
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
it('collapses dynamic path segments into one entry', () => {
|
|
95
|
+
const store = new RequestStore(outputDir);
|
|
96
|
+
|
|
97
|
+
store.addReadRequest(makeGet('/api/labels/8471', '', 'g1'));
|
|
98
|
+
store.addReadRequest(makeGet('/api/labels/9382', '', 'g2'));
|
|
99
|
+
|
|
100
|
+
expect(store.getCapturedRequests()).toHaveLength(1);
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
it('keeps variants that differ by query parameter names', () => {
|
|
104
|
+
const store = new RequestStore(outputDir);
|
|
105
|
+
|
|
106
|
+
store.addReadRequest(makeGet('/api/alpha-shop/tests', '', 'g1'));
|
|
107
|
+
store.addReadRequest(makeGet('/api/alpha-shop/tests', '?label=bug', 'g2'));
|
|
108
|
+
store.addReadRequest(makeGet('/api/alpha-shop/tests', '?label=urgent', 'g3'));
|
|
109
|
+
|
|
110
|
+
expect(store.getCapturedRequests()).toHaveLength(2);
|
|
111
|
+
});
|
|
112
|
+
|
|
113
|
+
it('does not write a response file for a bodiless capture', () => {
|
|
114
|
+
const store = new RequestStore(outputDir);
|
|
115
|
+
|
|
116
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g1'));
|
|
117
|
+
|
|
118
|
+
expect(existsSync(join(outputDir, 'requests', 'g1.request.yaml'))).toBe(true);
|
|
119
|
+
expect(existsSync(join(outputDir, 'requests', 'g1.response.json'))).toBe(false);
|
|
120
|
+
});
|
|
121
|
+
});
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Check the top of the file first: it must import `mkdtempSync`, `rmSync`, `existsSync` from `node:fs`, `tmpdir` from `node:os` and `join` from `node:path`. The existing `loadFromDisk` describe block already uses these, so add only what is missing.
|
|
125
|
+
|
|
126
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
127
|
+
|
|
128
|
+
```bash
|
|
129
|
+
cd /home/davert/projects/explorbot-fisherman-fetch
|
|
130
|
+
bun test tests/unit/request-store.test.ts
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Expected: FAIL — `store.addReadRequest is not a function`.
|
|
134
|
+
|
|
135
|
+
- [ ] **Step 3: Stop writing an empty response file**
|
|
136
|
+
|
|
137
|
+
In `src/api/request-result.ts`, the last two lines of `save()` currently read:
|
|
138
|
+
|
|
139
|
+
```ts
|
|
140
|
+
writeFileSync(this.requestFile, yaml, 'utf8');
|
|
141
|
+
writeFileSync(this.responseFile, this._rawResponseBody || '', 'utf8');
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Replace with:
|
|
145
|
+
|
|
146
|
+
```ts
|
|
147
|
+
writeFileSync(this.requestFile, yaml, 'utf8');
|
|
148
|
+
if (!this._rawResponseBody) return;
|
|
149
|
+
writeFileSync(this.responseFile, this._rawResponseBody, 'utf8');
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
The `rawResponseBody` getter already returns `''` when `responseFile` does not exist, so readers are unaffected.
|
|
153
|
+
|
|
154
|
+
- [ ] **Step 4: Add `addReadRequest` to RequestStore**
|
|
155
|
+
|
|
156
|
+
In `src/api/request-store.ts`, add a field beside the existing ones near the top of the class:
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
private readEndpointKeys = new Set<string>();
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
Add this public method directly after `addCapturedRequest`:
|
|
163
|
+
|
|
164
|
+
```ts
|
|
165
|
+
addReadRequest(result: RequestResult): void {
|
|
166
|
+
const key = readEndpointKey(result);
|
|
167
|
+
if (this.readEndpointKeys.has(key)) return;
|
|
168
|
+
this.readEndpointKeys.add(key);
|
|
169
|
+
this.capturedRequests.push(result);
|
|
170
|
+
result.save(this.outputDir);
|
|
171
|
+
}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Add this helper next to `normalizePathPattern` at the bottom of the file:
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
function readEndpointKey(result: RequestResult): string {
|
|
178
|
+
const query = result.fullUrl.split('?')[1] || '';
|
|
179
|
+
const names = [...new Set(new URLSearchParams(query).keys())].sort().join(',');
|
|
180
|
+
return `${result.method} ${normalizePathPattern(result.path)}?${names}`;
|
|
181
|
+
}
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
In `loadFromDisk`, register keys for read entries restored from disk so a second session does not re-add them. The loop body currently ends with `this.capturedRequests.push(result);` — add one line after it:
|
|
185
|
+
|
|
186
|
+
```ts
|
|
187
|
+
this.capturedRequests.push(result);
|
|
188
|
+
if (!result.isWrite) this.readEndpointKeys.add(readEndpointKey(result));
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Also clear the set in `clear()`, beside the three array resets:
|
|
192
|
+
|
|
193
|
+
```ts
|
|
194
|
+
this.readEndpointKeys.clear();
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
- [ ] **Step 5: Run the tests to verify they pass**
|
|
198
|
+
|
|
199
|
+
```bash
|
|
200
|
+
bun test tests/unit/request-store.test.ts
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
Expected: PASS, including every pre-existing test in the file.
|
|
204
|
+
|
|
205
|
+
- [ ] **Step 6: Write the failing XhrCapture tests**
|
|
206
|
+
|
|
207
|
+
Create `tests/unit/xhr-capture.test.ts`. `attach(page)` only registers a handler on `page.on('response', …)`, so a two-line fake page is enough to drive the real capture path:
|
|
208
|
+
|
|
209
|
+
```ts
|
|
210
|
+
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
|
211
|
+
import { mkdtempSync, rmSync } from 'node:fs';
|
|
212
|
+
import { tmpdir } from 'node:os';
|
|
213
|
+
import { join } from 'node:path';
|
|
214
|
+
import { RequestStore } from '../../src/api/request-store.ts';
|
|
215
|
+
import { XhrCapture } from '../../src/api/xhr-capture.ts';
|
|
216
|
+
|
|
217
|
+
function fakeResponse(opts: { method: string; url: string; status: number; contentType?: string; body?: string }) {
|
|
218
|
+
return {
|
|
219
|
+
request: () => ({
|
|
220
|
+
resourceType: () => 'xhr',
|
|
221
|
+
method: () => opts.method,
|
|
222
|
+
url: () => opts.url,
|
|
223
|
+
headers: () => ({ authorization: 'Bearer live' }),
|
|
224
|
+
postData: () => undefined,
|
|
225
|
+
}),
|
|
226
|
+
status: () => opts.status,
|
|
227
|
+
statusText: () => String(opts.status),
|
|
228
|
+
headers: () => ({ 'content-type': opts.contentType ?? 'application/json' }),
|
|
229
|
+
text: async () => opts.body ?? '{}',
|
|
230
|
+
};
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
describe('XhrCapture read endpoints', () => {
|
|
234
|
+
let outputDir: string;
|
|
235
|
+
let store: RequestStore;
|
|
236
|
+
let deliver: (response: any) => Promise<void>;
|
|
237
|
+
|
|
238
|
+
beforeEach(() => {
|
|
239
|
+
outputDir = mkdtempSync(join(tmpdir(), 'xhr-'));
|
|
240
|
+
store = new RequestStore(outputDir);
|
|
241
|
+
const capture = new XhrCapture(store, 'https://app.test');
|
|
242
|
+
capture.attach({
|
|
243
|
+
on: (_event: string, handler: any) => {
|
|
244
|
+
deliver = handler;
|
|
245
|
+
},
|
|
246
|
+
});
|
|
247
|
+
});
|
|
248
|
+
|
|
249
|
+
afterEach(() => {
|
|
250
|
+
rmSync(outputDir, { recursive: true, force: true });
|
|
251
|
+
});
|
|
252
|
+
|
|
253
|
+
it('records a successful JSON GET as a read endpoint without its body', async () => {
|
|
254
|
+
await deliver(fakeResponse({ method: 'GET', url: 'https://app.test/api/labels?page=1', status: 200 }));
|
|
255
|
+
|
|
256
|
+
const captured = store.getCapturedRequests();
|
|
257
|
+
expect(captured).toHaveLength(1);
|
|
258
|
+
expect(captured[0].method).toBe('GET');
|
|
259
|
+
expect(captured[0].fullUrl).toBe('/api/labels?page=1');
|
|
260
|
+
expect(captured[0].rawResponseBody).toBe('');
|
|
261
|
+
});
|
|
262
|
+
|
|
263
|
+
it('ignores a GET that did not return 200', async () => {
|
|
264
|
+
await deliver(fakeResponse({ method: 'GET', url: 'https://app.test/api/labels', status: 404 }));
|
|
265
|
+
|
|
266
|
+
expect(store.getCapturedRequests()).toHaveLength(0);
|
|
267
|
+
expect(store.getFailedRequests()).toHaveLength(1);
|
|
268
|
+
});
|
|
269
|
+
|
|
270
|
+
it('ignores a GET that did not return JSON', async () => {
|
|
271
|
+
await deliver(fakeResponse({ method: 'GET', url: 'https://app.test/api/labels', status: 200, contentType: 'text/html' }));
|
|
272
|
+
|
|
273
|
+
expect(store.getCapturedRequests()).toHaveLength(0);
|
|
274
|
+
});
|
|
275
|
+
|
|
276
|
+
it('still captures a write with its response body', async () => {
|
|
277
|
+
await deliver(fakeResponse({ method: 'POST', url: 'https://app.test/api/labels', status: 201, body: '{"id":1}' }));
|
|
278
|
+
|
|
279
|
+
const captured = store.getCapturedRequests();
|
|
280
|
+
expect(captured).toHaveLength(1);
|
|
281
|
+
expect(captured[0].method).toBe('POST');
|
|
282
|
+
expect(captured[0].rawResponseBody).toBe('{"id":1}');
|
|
283
|
+
});
|
|
284
|
+
});
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
- [ ] **Step 7: Run the tests to verify they fail**
|
|
288
|
+
|
|
289
|
+
```bash
|
|
290
|
+
bun test tests/unit/xhr-capture.test.ts
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
Expected: the two GET tests FAIL (nothing is captured); the 404 test and the write test already PASS.
|
|
294
|
+
|
|
295
|
+
- [ ] **Step 8: Capture GETs in XhrCapture**
|
|
296
|
+
|
|
297
|
+
In `src/api/xhr-capture.ts`, `captureResponse` currently drops every non-write before the content-type check. The failed-request block must stay first so non-JSON failures keep being recorded. Replace the section from `if (!WRITE_METHODS.has(method)) return;` through `if (status === 304) return;` — including the `const contentType` and `JSON_CONTENT_TYPES` lines sitting between them — with:
|
|
298
|
+
|
|
299
|
+
```ts
|
|
300
|
+
const contentType = response.headers()['content-type'] || '';
|
|
301
|
+
if (!JSON_CONTENT_TYPES.test(contentType)) return;
|
|
302
|
+
|
|
303
|
+
if (method === 'GET') {
|
|
304
|
+
if (status !== 200) return;
|
|
305
|
+
this.captureReadEndpoint(request, response);
|
|
306
|
+
return;
|
|
307
|
+
}
|
|
308
|
+
|
|
309
|
+
if (!WRITE_METHODS.has(method)) return;
|
|
310
|
+
|
|
311
|
+
if (status === 304) return;
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
Add this private method after `captureResponse`:
|
|
315
|
+
|
|
316
|
+
```ts
|
|
317
|
+
private captureReadEndpoint(request: any, response: any): void {
|
|
318
|
+
const parsedUrl = new URL(request.url());
|
|
319
|
+
|
|
320
|
+
const requestHeaders: Record<string, string> = {};
|
|
321
|
+
for (const [k, v] of Object.entries(request.headers())) {
|
|
322
|
+
requestHeaders[k] = String(v);
|
|
323
|
+
}
|
|
324
|
+
|
|
325
|
+
const result = new RequestResult({
|
|
326
|
+
id: generateRequestId('GET', parsedUrl.pathname, 'xhr_'),
|
|
327
|
+
method: 'GET',
|
|
328
|
+
path: parsedUrl.pathname,
|
|
329
|
+
fullUrl: parsedUrl.pathname + parsedUrl.search,
|
|
330
|
+
requestHeaders,
|
|
331
|
+
status: response.status(),
|
|
332
|
+
statusText: response.statusText(),
|
|
333
|
+
responseHeaders: {},
|
|
334
|
+
timing: 0,
|
|
335
|
+
timestamp: new Date(),
|
|
336
|
+
});
|
|
337
|
+
result.rawResponseBodyValue = '';
|
|
338
|
+
|
|
339
|
+
this.store.addReadRequest(result);
|
|
340
|
+
}
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
- [ ] **Step 9: Run the whole unit suite**
|
|
344
|
+
|
|
345
|
+
```bash
|
|
346
|
+
bun run format
|
|
347
|
+
bun test tests/unit/
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
Expected: PASS. `request-result.test.ts` and `request-store.test.ts` in particular must be green — the `save()` change touches every persisted request.
|
|
351
|
+
|
|
352
|
+
- [ ] **Step 10: Commit**
|
|
353
|
+
|
|
354
|
+
```bash
|
|
355
|
+
git add src/api/request-result.ts src/api/request-store.ts src/api/xhr-capture.ts tests/unit/request-store.test.ts tests/unit/xhr-capture.test.ts
|
|
356
|
+
git commit -m "feat: record successful GET endpoints as read captures"
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
---
|
|
360
|
+
|
|
361
|
+
### Task 2: Read and write endpoint families in RequestStore
|
|
362
|
+
|
|
363
|
+
**Files:**
|
|
364
|
+
- Modify: `src/api/request-store.ts:82-97` (`toEndpointList`), `src/api/request-store.ts:161-186` (`getWriteRequestsForScope`), file bottom (helpers and the `EndpointFamily` type)
|
|
365
|
+
- Test: `tests/unit/request-store.test.ts`
|
|
366
|
+
|
|
367
|
+
**Interfaces:**
|
|
368
|
+
- Consumes: `RequestStore.addReadRequest` and the read captures from Task 1; `RequestResult.isWrite`.
|
|
369
|
+
- Produces:
|
|
370
|
+
- `export type EndpointFamily = 'read' | 'write';` exported from `src/api/request-store.ts`.
|
|
371
|
+
- `RequestStore.toEndpointList(scopePath?: string, methods: EndpointFamily = 'write'): string` — one line per distinct endpoint. Write lines keep today's exact `METHOD /normalized/path` shape. Read lines append ` ?name1,name2` when the capture carried query parameters.
|
|
372
|
+
- `RequestStore.getReadRequestsForScope(scopePath: string): RequestResult[]` — the read counterpart of `getWriteRequestsForScope`, same scoping heuristic.
|
|
373
|
+
- `getWriteRequestsForScope` keeps its exact signature and behaviour. Task 4 calls `toEndpointList` with both families.
|
|
374
|
+
|
|
375
|
+
- [ ] **Step 1: Write the failing tests**
|
|
376
|
+
|
|
377
|
+
Append to the `describe('read endpoint capture')` block added in Task 1, reusing its `makeGet` helper:
|
|
378
|
+
|
|
379
|
+
```ts
|
|
380
|
+
it('lists read endpoints with their query parameter names, not values', () => {
|
|
381
|
+
const store = new RequestStore(outputDir);
|
|
382
|
+
|
|
383
|
+
store.addReadRequest(makeGet('/api/alpha-shop/tests', '?label=bug&page=2', 'g1'));
|
|
384
|
+
|
|
385
|
+
expect(store.toEndpointList(undefined, 'read')).toBe('GET /api/alpha-shop/tests ?label,page');
|
|
386
|
+
});
|
|
387
|
+
|
|
388
|
+
it('keeps read endpoints out of the write list and writes out of the read list', () => {
|
|
389
|
+
const store = new RequestStore(outputDir);
|
|
390
|
+
|
|
391
|
+
store.addCapturedRequest(makeRequest('POST', '/api/alpha-shop/suites', 201));
|
|
392
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g1'));
|
|
393
|
+
|
|
394
|
+
expect(store.toEndpointList()).toBe('POST /api/alpha-shop/suites');
|
|
395
|
+
expect(store.toEndpointList(undefined, 'read')).toBe('GET /api/alpha-shop/labels');
|
|
396
|
+
});
|
|
397
|
+
|
|
398
|
+
it('scopes read endpoints the same way write endpoints are scoped', () => {
|
|
399
|
+
const store = new RequestStore(outputDir);
|
|
400
|
+
|
|
401
|
+
store.addReadRequest(makeGet('/api/alpha-shop/labels', '', 'g1'));
|
|
402
|
+
store.addReadRequest(makeGet('/api/other-shop/labels', '', 'g2'));
|
|
403
|
+
|
|
404
|
+
const scoped = store.getReadRequestsForScope('/projects/alpha-shop/tests');
|
|
405
|
+
|
|
406
|
+
expect(scoped).toHaveLength(1);
|
|
407
|
+
expect(scoped[0].path).toBe('/api/alpha-shop/labels');
|
|
408
|
+
});
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
412
|
+
|
|
413
|
+
```bash
|
|
414
|
+
bun test tests/unit/request-store.test.ts
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
Expected: FAIL — `store.getReadRequestsForScope is not a function`, and `toEndpointList` ignores its second argument.
|
|
418
|
+
|
|
419
|
+
- [ ] **Step 3: Make `toEndpointList` family-aware**
|
|
420
|
+
|
|
421
|
+
Replace `toEndpointList` in `src/api/request-store.ts` with:
|
|
422
|
+
|
|
423
|
+
```ts
|
|
424
|
+
toEndpointList(scopePath?: string, methods: EndpointFamily = 'write'): string {
|
|
425
|
+
let requests = this.capturedRequests.filter((r) => matchesFamily(r, methods));
|
|
426
|
+
if (scopePath) requests = this.getRequestsForScope(scopePath, methods);
|
|
427
|
+
|
|
428
|
+
const seen = new Set<string>();
|
|
429
|
+
const lines: string[] = [];
|
|
430
|
+
|
|
431
|
+
for (const req of requests) {
|
|
432
|
+
const line = `${req.method} ${normalizePathPattern(req.path)}${queryParamHint(req)}`;
|
|
433
|
+
if (seen.has(line)) continue;
|
|
434
|
+
seen.add(line);
|
|
435
|
+
lines.push(line);
|
|
436
|
+
}
|
|
437
|
+
|
|
438
|
+
return lines.join('\n');
|
|
439
|
+
}
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
- [ ] **Step 4: Split the scoping heuristic across both families**
|
|
443
|
+
|
|
444
|
+
Replace `getWriteRequestsForScope` with two thin public methods, and move its body into one private method placed at the end of the class (after `clear()`), so private methods stay below public ones:
|
|
445
|
+
|
|
446
|
+
```ts
|
|
447
|
+
getWriteRequestsForScope(scopePath: string): RequestResult[] {
|
|
448
|
+
return this.getRequestsForScope(scopePath, 'write');
|
|
449
|
+
}
|
|
450
|
+
|
|
451
|
+
getReadRequestsForScope(scopePath: string): RequestResult[] {
|
|
452
|
+
return this.getRequestsForScope(scopePath, 'read');
|
|
453
|
+
}
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
```ts
|
|
457
|
+
private getRequestsForScope(scopePath: string, methods: EndpointFamily): RequestResult[] {
|
|
458
|
+
const candidates = this.capturedRequests.filter((r) => matchesFamily(r, methods));
|
|
459
|
+
const scopeSegments = scopePath.split('/').filter(Boolean);
|
|
460
|
+
if (scopeSegments.length === 0) return candidates;
|
|
461
|
+
|
|
462
|
+
let scoped: RequestResult[] = [];
|
|
463
|
+
let fewest = Number.POSITIVE_INFINITY;
|
|
464
|
+
let ambiguous = false;
|
|
465
|
+
for (const segment of scopeSegments) {
|
|
466
|
+
if (isDynamicSegment(segment)) continue;
|
|
467
|
+
const matches = candidates.filter((r) => r.path.split('/').includes(segment));
|
|
468
|
+
if (matches.length === 0 || matches.length > fewest) continue;
|
|
469
|
+
if (matches.length === fewest) {
|
|
470
|
+
if (!scoped.every((r, i) => r.id === matches[i].id)) ambiguous = true;
|
|
471
|
+
continue;
|
|
472
|
+
}
|
|
473
|
+
scoped = matches;
|
|
474
|
+
fewest = matches.length;
|
|
475
|
+
ambiguous = false;
|
|
476
|
+
}
|
|
477
|
+
if (ambiguous) return [];
|
|
478
|
+
|
|
479
|
+
return scoped;
|
|
480
|
+
}
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
The old local `const writeMethods = new Set([...])` line goes away — `matchesFamily` owns that knowledge now.
|
|
484
|
+
|
|
485
|
+
- [ ] **Step 5: Add the helpers and the type**
|
|
486
|
+
|
|
487
|
+
Beside `normalizePathPattern` and `readEndpointKey` at the bottom of `src/api/request-store.ts`:
|
|
488
|
+
|
|
489
|
+
```ts
|
|
490
|
+
function matchesFamily(result: RequestResult, methods: EndpointFamily): boolean {
|
|
491
|
+
if (methods === 'write') return result.isWrite;
|
|
492
|
+
return result.method === 'GET';
|
|
493
|
+
}
|
|
494
|
+
|
|
495
|
+
function queryParamHint(result: RequestResult): string {
|
|
496
|
+
if (result.isWrite) return '';
|
|
497
|
+
const query = result.fullUrl.split('?')[1];
|
|
498
|
+
if (!query) return '';
|
|
499
|
+
const names = [...new Set(new URLSearchParams(query).keys())].sort();
|
|
500
|
+
if (names.length === 0) return '';
|
|
501
|
+
return ` ?${names.join(',')}`;
|
|
502
|
+
}
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
At the very end of the file:
|
|
506
|
+
|
|
507
|
+
```ts
|
|
508
|
+
export type EndpointFamily = 'read' | 'write';
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
- [ ] **Step 6: Run the tests to verify they pass**
|
|
512
|
+
|
|
513
|
+
```bash
|
|
514
|
+
bun run format
|
|
515
|
+
bun test tests/unit/request-store.test.ts
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
Expected: PASS. The pre-existing assertions `toEndpointList()` → `'PATCH /api/suites/{id}'` and `toEndpointList('/projects/alpha-shop')` → `'POST /api/alpha-shop/suites'` must still hold — the `'write'` default preserves them.
|
|
519
|
+
|
|
520
|
+
- [ ] **Step 7: Commit**
|
|
521
|
+
|
|
522
|
+
```bash
|
|
523
|
+
git add src/api/request-store.ts tests/unit/request-store.test.ts
|
|
524
|
+
git commit -m "feat: split read and write endpoint families in RequestStore"
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
---
|
|
528
|
+
|
|
529
|
+
### Task 3: Read-only mode for Fisherman tools
|
|
530
|
+
|
|
531
|
+
**Files:**
|
|
532
|
+
- Modify: `src/ai/fisherman-tools.ts` — the `createFishermanTools` signature, the `request` and `getEndpointSpec` method enums, the `request` success return, a second `finish` tool, and `synthesizeResult`
|
|
533
|
+
- Test: `tests/unit/fisherman-tools.test.ts`
|
|
534
|
+
|
|
535
|
+
**Interfaces:**
|
|
536
|
+
- Consumes: `ApiClient.request`, `RequestStore.getMadeRequests` / `addMadeRequest`, `RequestResult.isWrite` / `status` / `error` / `rawResponseBody`.
|
|
537
|
+
- Produces: `createFishermanTools(apiClient, requestStore, opts: { spec?: any; baseEndpoint?: string; readOnly?: boolean })`. With `readOnly: true`:
|
|
538
|
+
- `tools.request` accepts only `method: 'GET'` and returns `{ success, status, extracted, bodyPreview }` on 2xx, where `bodyPreview` is the raw body truncated to 2000 characters.
|
|
539
|
+
- `tools.finish` takes `{ answer: string }` and succeeds when at least one successful GET was made in this run, producing `{ success: true, summary: answer, created: [], failed: [] }`.
|
|
540
|
+
- `getResult()` / `finishFromText()` measure success against successful GETs instead of successful writes, and never populate `created`.
|
|
541
|
+
- The returned object's type is `Record<string, any>` so both `finish` shapes fit. Task 4 passes it straight to `provider.invokeConversation`.
|
|
542
|
+
|
|
543
|
+
- [ ] **Step 1: Write the failing tests**
|
|
544
|
+
|
|
545
|
+
Append to `tests/unit/fisherman-tools.test.ts`. It needs a read counterpart to the file's existing `madeWrite` helper — add `madeRead` beside it at the bottom of the file:
|
|
546
|
+
|
|
547
|
+
```ts
|
|
548
|
+
function madeRead(path: string, status: number, body = '[]'): any {
|
|
549
|
+
return {
|
|
550
|
+
method: 'GET',
|
|
551
|
+
path,
|
|
552
|
+
status,
|
|
553
|
+
error: undefined,
|
|
554
|
+
isWrite: false,
|
|
555
|
+
rawResponseBody: body,
|
|
556
|
+
responseBody: JSON.parse(body),
|
|
557
|
+
statusText: String(status),
|
|
558
|
+
extractIdAndTitle: () => ({}),
|
|
559
|
+
toEndpoint: () => `GET ${path}`,
|
|
560
|
+
toSummary: () => `GET ${path} → ${status} (0ms)`,
|
|
561
|
+
};
|
|
562
|
+
}
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
Add this describe block at the end of the file:
|
|
566
|
+
|
|
567
|
+
```ts
|
|
568
|
+
describe('Fisherman read-only tools', () => {
|
|
569
|
+
it('offers no write method on the request tool', () => {
|
|
570
|
+
const { tools } = createFishermanTools({} as any, store(), { readOnly: true });
|
|
571
|
+
|
|
572
|
+
const methods = tools.request.inputSchema.shape.method.options;
|
|
573
|
+
|
|
574
|
+
expect(methods).toEqual(['GET']);
|
|
575
|
+
});
|
|
576
|
+
|
|
577
|
+
it('returns a body preview so the answer can quote real values', async () => {
|
|
578
|
+
const labels = madeRead('/api/labels', 200, '[{"id":1,"name":"Bug"}]');
|
|
579
|
+
const apiClient = { request: async () => labels };
|
|
580
|
+
const { tools } = createFishermanTools(apiClient as any, store(), { readOnly: true });
|
|
581
|
+
|
|
582
|
+
const result: any = await tools.request.execute({ method: 'GET', path: '/api/labels' }, {} as any);
|
|
583
|
+
|
|
584
|
+
expect(result.success).toBe(true);
|
|
585
|
+
expect(result.bodyPreview).toBe('[{"id":1,"name":"Bug"}]');
|
|
586
|
+
});
|
|
587
|
+
|
|
588
|
+
it('accepts a finish carrying the answer once a read succeeded', async () => {
|
|
589
|
+
const made: any[] = [];
|
|
590
|
+
const apiClient = { request: async () => madeRead('/api/labels', 200) };
|
|
591
|
+
const { tools, getResult } = createFishermanTools(apiClient as any, store(undefined, made), { readOnly: true });
|
|
592
|
+
|
|
593
|
+
await tools.request.execute({ method: 'GET', path: '/api/labels' }, {} as any);
|
|
594
|
+
const finished: any = await tools.finish.execute({ answer: 'No labels exist yet' }, {} as any);
|
|
595
|
+
|
|
596
|
+
expect(finished.finished).toBe(true);
|
|
597
|
+
expect(getResult()).toEqual({ success: true, summary: 'No labels exist yet', created: [], failed: [] });
|
|
598
|
+
});
|
|
599
|
+
|
|
600
|
+
it('rejects a finish when no read succeeded', async () => {
|
|
601
|
+
const { tools } = createFishermanTools({} as any, store(), { readOnly: true });
|
|
602
|
+
|
|
603
|
+
const finished: any = await tools.finish.execute({ answer: 'Three labels exist' }, {} as any);
|
|
604
|
+
|
|
605
|
+
expect(finished.finished).toBe(false);
|
|
606
|
+
});
|
|
607
|
+
|
|
608
|
+
it('reports no created items when a read run ends without finishing', async () => {
|
|
609
|
+
const made: any[] = [];
|
|
610
|
+
const apiClient = { request: async () => madeRead('/api/labels', 200) };
|
|
611
|
+
const { tools, getResult, finishFromText } = createFishermanTools(apiClient as any, store(undefined, made), { readOnly: true });
|
|
612
|
+
|
|
613
|
+
await tools.request.execute({ method: 'GET', path: '/api/labels' }, {} as any);
|
|
614
|
+
finishFromText('One label exists');
|
|
615
|
+
|
|
616
|
+
const result = getResult();
|
|
617
|
+
expect(result.success).toBe(true);
|
|
618
|
+
expect(result.summary).toBe('One label exists');
|
|
619
|
+
expect(result.created).toEqual([]);
|
|
620
|
+
});
|
|
621
|
+
});
|
|
622
|
+
```
|
|
623
|
+
|
|
624
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
625
|
+
|
|
626
|
+
```bash
|
|
627
|
+
bun test tests/unit/fisherman-tools.test.ts
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
Expected: FAIL — the `request` tool still lists all five methods and `finish` still demands `created`.
|
|
631
|
+
|
|
632
|
+
The first test reaches into `tools.request.inputSchema.shape.method.options`, which assumes the AI SDK leaves the Zod object unwrapped on `inputSchema`. If it fails for that reason rather than for the enum's contents, assert on the serialized schema instead: `expect(JSON.stringify(tools.request.inputSchema)).not.toContain('DELETE')`.
|
|
633
|
+
|
|
634
|
+
- [ ] **Step 3: Thread the flag and the success predicate through the factory**
|
|
635
|
+
|
|
636
|
+
In `src/ai/fisherman-tools.ts`, widen the signature and derive the mode-dependent pieces at the top of the function:
|
|
637
|
+
|
|
638
|
+
```ts
|
|
639
|
+
export function createFishermanTools(apiClient: ApiClient, requestStore: RequestStore, opts: { spec?: any; baseEndpoint?: string; readOnly?: boolean }) {
|
|
640
|
+
const readOnly = opts.readOnly === true;
|
|
641
|
+
let finished = false;
|
|
642
|
+
let result: FishermanResult | null = null;
|
|
643
|
+
const ledgerStart = requestStore.getMadeRequests().length;
|
|
644
|
+
|
|
645
|
+
let allowedMethods: string[] = ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'];
|
|
646
|
+
if (readOnly) allowedMethods = ['GET'];
|
|
647
|
+
|
|
648
|
+
const runRequests = () => requestStore.getMadeRequests().slice(ledgerStart);
|
|
649
|
+
const successfulWrites = () => runRequests().filter((r) => r.isWrite && !r.error && r.status >= 200 && r.status < 400);
|
|
650
|
+
const successfulReads = () => runRequests().filter((r) => !r.isWrite && !r.error && r.status >= 200 && r.status < 400);
|
|
651
|
+
const succeeded = () => {
|
|
652
|
+
if (readOnly) return successfulReads();
|
|
653
|
+
return successfulWrites();
|
|
654
|
+
};
|
|
655
|
+
const getResult = () => result ?? synthesizeResult(runRequests(), succeeded(), false, readOnly);
|
|
656
|
+
const isFinished = () => finished;
|
|
657
|
+
const finishFromText = (text?: string) => {
|
|
658
|
+
finished = true;
|
|
659
|
+
const synthesized = synthesizeResult(runRequests(), succeeded(), true, readOnly);
|
|
660
|
+
if (text && synthesized.success) synthesized.summary = text;
|
|
661
|
+
result = synthesized;
|
|
662
|
+
};
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
Use `allowedMethods` in both schemas. In `getEndpointSpec` and `request`, replace `z.enum(['GET', 'POST', 'PUT', 'PATCH', 'DELETE'])` with:
|
|
666
|
+
|
|
667
|
+
```ts
|
|
668
|
+
method: z.enum(allowedMethods as [string, ...string[]]).describe('HTTP method'),
|
|
669
|
+
```
|
|
670
|
+
|
|
671
|
+
- [ ] **Step 4: Return a body preview in read mode**
|
|
672
|
+
|
|
673
|
+
In the `request` tool's `execute`, replace the 2xx return block:
|
|
674
|
+
|
|
675
|
+
```ts
|
|
676
|
+
const extracted = extractKeyFields(reqResult.responseBody);
|
|
677
|
+
tag('success').log(`Fisherman: ${input.method} ${input.path} > ${statusLine}`);
|
|
678
|
+
const output: Record<string, any> = {
|
|
679
|
+
success: true,
|
|
680
|
+
status: reqResult.status,
|
|
681
|
+
extracted,
|
|
682
|
+
};
|
|
683
|
+
if (readOnly) output.bodyPreview = reqResult.rawResponseBody.substring(0, BODY_PREVIEW_LIMIT);
|
|
684
|
+
return output;
|
|
685
|
+
```
|
|
686
|
+
|
|
687
|
+
Add the constant next to the other module constants at the top of the file:
|
|
688
|
+
|
|
689
|
+
```ts
|
|
690
|
+
const BODY_PREVIEW_LIMIT = 2000;
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
- [ ] **Step 5: Add the read-mode finish and select between the two**
|
|
694
|
+
|
|
695
|
+
Keep the existing `finish` tool exactly as it is, but bind it to a name. Change `const tools = {` into named consts so both finish shapes can be built, then assemble:
|
|
696
|
+
|
|
697
|
+
```ts
|
|
698
|
+
const finishWrite = tool({
|
|
699
|
+
// the existing finish tool, unchanged, except its guard now reads:
|
|
700
|
+
// const writes = succeeded();
|
|
701
|
+
});
|
|
702
|
+
|
|
703
|
+
const finishRead = tool({
|
|
704
|
+
description: 'Report the answer to the question. Call when the requests have shown what exists.',
|
|
705
|
+
inputSchema: z.object({
|
|
706
|
+
answer: z.string().describe('What the data shows, quoting the concrete names, titles and ids that were returned'),
|
|
707
|
+
}),
|
|
708
|
+
execute: async ({ answer }) => {
|
|
709
|
+
if (succeeded().length === 0) {
|
|
710
|
+
tag('warning').log('Fisherman: finish rejected — no successful request in this run');
|
|
711
|
+
return { finished: false, error: 'No successful request was made in this run, so nothing was read. Keep working, or call stop if the question cannot be answered.' };
|
|
712
|
+
}
|
|
713
|
+
|
|
714
|
+
tag('success').log(`Fisherman answered: ${answer}`);
|
|
715
|
+
finished = true;
|
|
716
|
+
result = { success: true, summary: answer, created: [], failed: [] };
|
|
717
|
+
return { finished: true };
|
|
718
|
+
},
|
|
719
|
+
});
|
|
720
|
+
|
|
721
|
+
const tools: Record<string, any> = { getEndpointSpec, request, finish: finishWrite, stop };
|
|
722
|
+
if (readOnly) tools.finish = finishRead;
|
|
723
|
+
|
|
724
|
+
return { tools, getResult, isFinished, finishFromText };
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
Extract `getEndpointSpec`, `request` and `stop` from the current object literal into `const` declarations above this block, unchanged apart from the enum edit in Step 3 and the preview in Step 4. Inside `finishWrite`, the line `const writes = successfulWrites();` becomes `const writes = succeeded();` so the two paths share one predicate — in write mode the two are identical.
|
|
728
|
+
|
|
729
|
+
- [ ] **Step 6: Make `synthesizeResult` family-aware**
|
|
730
|
+
|
|
731
|
+
```ts
|
|
732
|
+
function synthesizeResult(made: RequestResult[], succeeded: RequestResult[], declaredDone: boolean, readOnly: boolean): FishermanResult {
|
|
733
|
+
const failures = made.filter((r) => r.status >= 400 || r.error);
|
|
734
|
+
let summary = `Stopped before finishing: ${made.length} requests, ${succeeded.length} successful, ${failures.length} failed`;
|
|
735
|
+
const lastFailure = failures[failures.length - 1];
|
|
736
|
+
if (lastFailure) summary += `; last failure: ${lastFailure.toSummary()}`;
|
|
737
|
+
|
|
738
|
+
const result: FishermanResult = { success: declaredDone && succeeded.length > 0, summary, created: [], failed: [] };
|
|
739
|
+
if (!readOnly) result.created = succeeded.map(toCreatedItem);
|
|
740
|
+
return result;
|
|
741
|
+
}
|
|
742
|
+
```
|
|
743
|
+
|
|
744
|
+
- [ ] **Step 7: Run the tests to verify they pass**
|
|
745
|
+
|
|
746
|
+
```bash
|
|
747
|
+
bun run format
|
|
748
|
+
bun test tests/unit/fisherman-tools.test.ts
|
|
749
|
+
```
|
|
750
|
+
|
|
751
|
+
Expected: PASS, including every pre-existing test in the file.
|
|
752
|
+
|
|
753
|
+
- [ ] **Step 8: Commit**
|
|
754
|
+
|
|
755
|
+
```bash
|
|
756
|
+
git add src/ai/fisherman-tools.ts tests/unit/fisherman-tools.test.ts
|
|
757
|
+
git commit -m "feat: add read-only mode to Fisherman tools"
|
|
758
|
+
```
|
|
759
|
+
|
|
760
|
+
---
|
|
761
|
+
|
|
762
|
+
### Task 4: Fisherman.lookupData and the shared session loop
|
|
763
|
+
|
|
764
|
+
**Files:**
|
|
765
|
+
- Modify: `src/ai/fisherman.ts` — extract `runSession`, add `lookupData` and `buildLookupSystemPrompt`, make `buildEndpointList` family-aware, drop the `mode = 'disabled'` mutation in `prepareData`
|
|
766
|
+
- Test: `tests/integration/fisherman.test.ts`
|
|
767
|
+
|
|
768
|
+
**Interfaces:**
|
|
769
|
+
- Consumes: `createFishermanTools(..., { readOnly: true })` from Task 3; `RequestStore.toEndpointList(scopePath, methods)` from Task 2; existing `Provider.startConversation` / `invokeConversation`, `loop` from `src/utils/loop.ts`, `listAllEndpoints` from `src/api/spec-reader.ts`.
|
|
770
|
+
- Produces: `Fisherman.lookupData(question: string, scopeUrl?: string, sessionName?: string): Promise<FishermanResult>` — on success `summary` holds the model's prose answer and `created` is empty. Task 5's Pilot tool calls exactly this signature.
|
|
771
|
+
|
|
772
|
+
- [ ] **Step 1: Write the failing tests**
|
|
773
|
+
|
|
774
|
+
Append to `tests/integration/fisherman.test.ts`, reusing its `requestResult`, `toolCall`, `extractPromptText` helpers and the `createFisherman` factory:
|
|
775
|
+
|
|
776
|
+
```ts
|
|
777
|
+
it('answers a question from a read endpoint and never offers a write method', async () => {
|
|
778
|
+
const labels = requestResult('made_read_1', 'GET', '/api/alpha-shop/labels', 200);
|
|
779
|
+
labels.rawResponseBodyValue = JSON.stringify([{ id: 1, name: 'Bug' }, { id: 2, name: 'Urgent' }]);
|
|
780
|
+
apiResponses.push(labels);
|
|
781
|
+
|
|
782
|
+
requestStore.addReadRequest(readResult('xhr_100_GET_api_alpha-shop_labels', '/api/alpha-shop/labels'));
|
|
783
|
+
|
|
784
|
+
mock.on({ sequenceIndex: 0 }, { toolCalls: [toolCall('r1', 'request', { method: 'GET', path: '/api/alpha-shop/labels' })] });
|
|
785
|
+
mock.on({ sequenceIndex: 1 }, { toolCalls: [toolCall('r2', 'finish', { answer: 'Two labels exist: Bug and Urgent' })] });
|
|
786
|
+
mock.on({}, { content: 'done' });
|
|
787
|
+
|
|
788
|
+
const result = await createFisherman().lookupData('which labels exist?', '/projects/alpha-shop/tests');
|
|
789
|
+
|
|
790
|
+
expect(result.success).toBe(true);
|
|
791
|
+
expect(result.summary).toBe('Two labels exist: Bug and Urgent');
|
|
792
|
+
expect(result.created).toEqual([]);
|
|
793
|
+
|
|
794
|
+
const systemPrompt = extractPromptText(mock.getRequests()[0]);
|
|
795
|
+
expect(systemPrompt).toContain('GET /api/alpha-shop/labels');
|
|
796
|
+
|
|
797
|
+
const offeredTools = JSON.stringify(mock.getRequests()[0]?.body?.tools);
|
|
798
|
+
expect(offeredTools).toContain('GET');
|
|
799
|
+
expect(offeredTools).not.toContain('DELETE');
|
|
800
|
+
});
|
|
801
|
+
|
|
802
|
+
it('reports honestly when no read endpoint is known', async () => {
|
|
803
|
+
const result = await createFisherman().lookupData('which labels exist?', '/projects/alpha-shop/tests');
|
|
804
|
+
|
|
805
|
+
expect(result.success).toBe(false);
|
|
806
|
+
expect(mock.getRequests()).toHaveLength(0);
|
|
807
|
+
});
|
|
808
|
+
|
|
809
|
+
it('stays available after preparing data found no write endpoints', async () => {
|
|
810
|
+
requestStore = new RequestStore(outputDir);
|
|
811
|
+
requestStore.addReadRequest(readResult('xhr_200_GET_api_alpha-shop_labels', '/api/alpha-shop/labels'));
|
|
812
|
+
|
|
813
|
+
const fisherman = createFisherman();
|
|
814
|
+
const result = await fisherman.prepareData('1 label', '/projects/alpha-shop/tests');
|
|
815
|
+
|
|
816
|
+
expect(result.success).toBe(false);
|
|
817
|
+
expect(fisherman.isAvailable()).toBe(true);
|
|
818
|
+
expect(mock.getRequests()).toHaveLength(0);
|
|
819
|
+
});
|
|
820
|
+
```
|
|
821
|
+
|
|
822
|
+
The third test reassigns `requestStore` before `createFisherman()` reads it, so the two POST captures the `beforeEach` seeds are out of the way and the write endpoint list really is empty — which is what the removed `this.mode = 'disabled'` mutation used to poison.
|
|
823
|
+
|
|
824
|
+
Add this helper next to `requestResult` at the top of the file:
|
|
825
|
+
|
|
826
|
+
```ts
|
|
827
|
+
function readResult(id: string, urlPath: string, search = ''): RequestResult {
|
|
828
|
+
const result = new RequestResult({
|
|
829
|
+
id,
|
|
830
|
+
method: 'GET',
|
|
831
|
+
path: urlPath,
|
|
832
|
+
fullUrl: `${urlPath}${search}`,
|
|
833
|
+
requestHeaders: {},
|
|
834
|
+
status: 200,
|
|
835
|
+
statusText: '200',
|
|
836
|
+
responseHeaders: {},
|
|
837
|
+
timing: 0,
|
|
838
|
+
timestamp: new Date(),
|
|
839
|
+
});
|
|
840
|
+
result.rawResponseBodyValue = '';
|
|
841
|
+
return result;
|
|
842
|
+
}
|
|
843
|
+
```
|
|
844
|
+
|
|
845
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
846
|
+
|
|
847
|
+
```bash
|
|
848
|
+
bun test tests/integration/fisherman.test.ts
|
|
849
|
+
```
|
|
850
|
+
|
|
851
|
+
Expected: FAIL — `fisherman.lookupData is not a function`.
|
|
852
|
+
|
|
853
|
+
- [ ] **Step 3: Extract the session loop**
|
|
854
|
+
|
|
855
|
+
In `src/ai/fisherman.ts`, add `import type { Conversation } from './conversation.ts';` to the imports.
|
|
856
|
+
|
|
857
|
+
Add this private method after the existing public methods, before `detectMode`:
|
|
858
|
+
|
|
859
|
+
```ts
|
|
860
|
+
private async runSession(conversation: Conversation, tools: Record<string, any>, opts: { isFinished: () => boolean; finishFromText: (text?: string) => void; label: string }): Promise<void> {
|
|
861
|
+
const ledgerStart = this.requestStore.getMadeRequests().length;
|
|
862
|
+
|
|
863
|
+
await loop(
|
|
864
|
+
async ({ stop, iteration }) => {
|
|
865
|
+
debugLog(`iteration ${iteration}`);
|
|
866
|
+
const invokeResult = await this.provider.invokeConversation(conversation, tools, {
|
|
867
|
+
maxToolRoundtrips: MAX_TOOL_ROUNDTRIPS,
|
|
868
|
+
agentName: 'fisherman',
|
|
869
|
+
});
|
|
870
|
+
debugLog(`iteration ${iteration} done, text: ${invokeResult?.response?.text?.slice(0, 200) || '(none)'}`);
|
|
871
|
+
|
|
872
|
+
if (opts.isFinished()) {
|
|
873
|
+
stop();
|
|
874
|
+
return;
|
|
875
|
+
}
|
|
876
|
+
|
|
877
|
+
if (!invokeResult?.toolExecutions?.length) {
|
|
878
|
+
debugLog('no tool call in this turn — treating as finish');
|
|
879
|
+
opts.finishFromText(invokeResult?.response?.text);
|
|
880
|
+
stop();
|
|
881
|
+
return;
|
|
882
|
+
}
|
|
883
|
+
|
|
884
|
+
if (this.isStuckOnEndpoint(ledgerStart)) {
|
|
885
|
+
tag('warning').log('Fisherman: repeated failures on the same endpoint — stopping');
|
|
886
|
+
stop();
|
|
887
|
+
return;
|
|
888
|
+
}
|
|
889
|
+
|
|
890
|
+
if (iteration >= MAX_ITERATIONS) {
|
|
891
|
+
tag('warning').log('Fisherman: max iterations reached');
|
|
892
|
+
stop();
|
|
893
|
+
}
|
|
894
|
+
},
|
|
895
|
+
{
|
|
896
|
+
maxAttempts: MAX_ITERATIONS,
|
|
897
|
+
observability: {
|
|
898
|
+
name: opts.label,
|
|
899
|
+
agent: 'fisherman',
|
|
900
|
+
sessionId: this.sessionName,
|
|
901
|
+
},
|
|
902
|
+
catch: async ({ error, stop }) => {
|
|
903
|
+
debugLog(`error: ${error.message}`);
|
|
904
|
+
tag('warning').log(`Fisherman error: ${error.message}`);
|
|
905
|
+
stop();
|
|
906
|
+
},
|
|
907
|
+
}
|
|
908
|
+
);
|
|
909
|
+
}
|
|
910
|
+
```
|
|
911
|
+
|
|
912
|
+
In `prepareData`, delete the local `const ledgerStart = ...` line and the whole `await loop(...)` call, replacing them with:
|
|
913
|
+
|
|
914
|
+
```ts
|
|
915
|
+
await this.runSession(conversation, tools, { isFinished, finishFromText, label: `fisherman: ${instructions.slice(0, 50)}` });
|
|
916
|
+
```
|
|
917
|
+
|
|
918
|
+
In the same method, drop the `this.mode = 'disabled';` line from the empty-endpoint-list branch, leaving:
|
|
919
|
+
|
|
920
|
+
```ts
|
|
921
|
+
if (!endpointList) {
|
|
922
|
+
tag('warning').log('Fisherman: no endpoints available');
|
|
923
|
+
return { success: false, summary: 'No API endpoints available', created: [], failed: [] };
|
|
924
|
+
}
|
|
925
|
+
```
|
|
926
|
+
|
|
927
|
+
- [ ] **Step 4: Make the endpoint list family-aware**
|
|
928
|
+
|
|
929
|
+
Replace `buildEndpointList`:
|
|
930
|
+
|
|
931
|
+
```ts
|
|
932
|
+
private buildEndpointList(scopeUrl?: string, family: EndpointFamily = 'write'): string {
|
|
933
|
+
this.scopeDegraded = false;
|
|
934
|
+
if (this.mode === 'achieve' && this.spec) {
|
|
935
|
+
let specEndpoints = listAllEndpoints(this.spec, this.baseEndpoint);
|
|
936
|
+
if (family === 'read') specEndpoints = keepReadLines(specEndpoints);
|
|
937
|
+
if (specEndpoints) return specEndpoints;
|
|
938
|
+
}
|
|
939
|
+
|
|
940
|
+
const scoped = this.requestStore.toEndpointList(scopeUrl || '/', family);
|
|
941
|
+
if (scoped) return scoped;
|
|
942
|
+
|
|
943
|
+
this.scopeDegraded = true;
|
|
944
|
+
return this.requestStore.toEndpointList(undefined, family);
|
|
945
|
+
}
|
|
946
|
+
```
|
|
947
|
+
|
|
948
|
+
Import the type — `import { type EndpointFamily, ... } from '../api/request-store.ts';` — noting `RequestStore` is currently imported there as a type-only import, so it becomes `import type { EndpointFamily, RequestStore } from '../api/request-store.ts';`.
|
|
949
|
+
|
|
950
|
+
Add this module function at the end of the file:
|
|
951
|
+
|
|
952
|
+
```ts
|
|
953
|
+
function keepReadLines(endpointList: string): string {
|
|
954
|
+
return endpointList
|
|
955
|
+
.split('\n')
|
|
956
|
+
.filter((line) => line.startsWith('GET '))
|
|
957
|
+
.join('\n');
|
|
958
|
+
}
|
|
959
|
+
```
|
|
960
|
+
|
|
961
|
+
`getEndpointList(scopeUrl?)` — the public method Planner calls — keeps calling `buildEndpointList(scopeUrl)` and so keeps returning write endpoints. Do not change it.
|
|
962
|
+
|
|
963
|
+
- [ ] **Step 5: Add `lookupData`**
|
|
964
|
+
|
|
965
|
+
Add this public method directly after `prepareData`:
|
|
966
|
+
|
|
967
|
+
```ts
|
|
968
|
+
async lookupData(question: string, scopeUrl?: string, sessionName?: string): Promise<FishermanResult> {
|
|
969
|
+
this.sessionName = sessionName;
|
|
970
|
+
tag('info').log(`Fisherman [read]: ${question}`);
|
|
971
|
+
|
|
972
|
+
await this.ensureReady(scopeUrl);
|
|
973
|
+
|
|
974
|
+
if (this.mode === 'disabled') {
|
|
975
|
+
debugLog('disabled — no data for scope');
|
|
976
|
+
return { success: false, summary: 'No API data available for this scope', created: [], failed: [] };
|
|
977
|
+
}
|
|
978
|
+
|
|
979
|
+
const endpointList = this.buildEndpointList(scopeUrl, 'read');
|
|
980
|
+
debugLog(`read endpoints:\n${endpointList || '(none)'}`);
|
|
981
|
+
|
|
982
|
+
if (!endpointList) {
|
|
983
|
+
tag('warning').log('Fisherman: no read endpoints available');
|
|
984
|
+
return { success: false, summary: 'No read endpoints are known for this scope', created: [], failed: [] };
|
|
985
|
+
}
|
|
986
|
+
|
|
987
|
+
await this.refreshAuth();
|
|
988
|
+
|
|
989
|
+
const { tools, getResult, isFinished, finishFromText } = createFishermanTools(this.apiClient, this.requestStore, {
|
|
990
|
+
spec: this.spec,
|
|
991
|
+
baseEndpoint: this.baseEndpoint,
|
|
992
|
+
readOnly: true,
|
|
993
|
+
});
|
|
994
|
+
|
|
995
|
+
const conversation = this.provider.startConversation(this.buildLookupSystemPrompt(endpointList, Object.keys(tools), scopeUrl), 'fisherman');
|
|
996
|
+
conversation.addUserText(dedent`
|
|
997
|
+
Answer this question about data that already exists:
|
|
998
|
+
|
|
999
|
+
${question}
|
|
1000
|
+
|
|
1001
|
+
Make the requests needed to answer it, then call finish with the answer.
|
|
1002
|
+
If the available endpoints cannot answer it, call stop with the reason.
|
|
1003
|
+
`);
|
|
1004
|
+
|
|
1005
|
+
await this.runSession(conversation, tools, { isFinished, finishFromText, label: `fisherman lookup: ${question.slice(0, 50)}` });
|
|
1006
|
+
|
|
1007
|
+
const result = getResult();
|
|
1008
|
+
tag('info').log(`Fisherman answer: ${result.summary}`);
|
|
1009
|
+
return result;
|
|
1010
|
+
}
|
|
1011
|
+
```
|
|
1012
|
+
|
|
1013
|
+
- [ ] **Step 6: Add the lookup system prompt**
|
|
1014
|
+
|
|
1015
|
+
Add this private method after `buildSystemPrompt`:
|
|
1016
|
+
|
|
1017
|
+
```ts
|
|
1018
|
+
private buildLookupSystemPrompt(endpointList: string, toolNames: string[], scopeUrl?: string): string {
|
|
1019
|
+
let scopeBlock = '';
|
|
1020
|
+
if (scopeUrl) {
|
|
1021
|
+
scopeBlock = `\n\nSCOPE: You are answering about ${scopeUrl}.`;
|
|
1022
|
+
if (this.scopeDegraded) scopeBlock += '\nThe endpoint list could not be narrowed to this scope and may include endpoints belonging to other scopes. Prefer the endpoint whose path belongs to this scope.';
|
|
1023
|
+
}
|
|
1024
|
+
|
|
1025
|
+
return dedent`
|
|
1026
|
+
You are Fisherman — reading the API to report what data already exists. You change nothing.
|
|
1027
|
+
|
|
1028
|
+
AVAILABLE ENDPOINTS:
|
|
1029
|
+
${endpointList}
|
|
1030
|
+
${scopeBlock}
|
|
1031
|
+
|
|
1032
|
+
AVAILABLE TOOLS:
|
|
1033
|
+
${toolNames.join(', ')}.
|
|
1034
|
+
Use tool names exactly as listed. Do not invent aliases or combined names.
|
|
1035
|
+
Match each tool input schema exactly. Do not invent parameter names or pass extra fields.
|
|
1036
|
+
|
|
1037
|
+
WORKFLOW:
|
|
1038
|
+
1. Pick the endpoint that lists the kind of item the question is about
|
|
1039
|
+
2. Request it, and when the answer needs a parent resource, request the parent first and use its id
|
|
1040
|
+
3. Call finish with the answer, quoting the concrete names, titles and ids the responses returned
|
|
1041
|
+
|
|
1042
|
+
RULES:
|
|
1043
|
+
- Report only what a response actually returned. Never describe data you did not read
|
|
1044
|
+
- Report an empty collection as empty. An absent item must not be reported as present
|
|
1045
|
+
- Answer the question that was asked and stop. Do not survey unrelated endpoints
|
|
1046
|
+
- Use the response category and error text to correct a failed request. Retry a temporary or server failure once
|
|
1047
|
+
`;
|
|
1048
|
+
}
|
|
1049
|
+
```
|
|
1050
|
+
|
|
1051
|
+
- [ ] **Step 7: Run the tests to verify they pass**
|
|
1052
|
+
|
|
1053
|
+
```bash
|
|
1054
|
+
bun run format
|
|
1055
|
+
bun test tests/integration/fisherman.test.ts
|
|
1056
|
+
```
|
|
1057
|
+
|
|
1058
|
+
Expected: PASS, including the five pre-existing `prepareData` tests — the extracted `runSession` must not change their behaviour.
|
|
1059
|
+
|
|
1060
|
+
- [ ] **Step 8: Commit**
|
|
1061
|
+
|
|
1062
|
+
```bash
|
|
1063
|
+
git add src/ai/fisherman.ts tests/integration/fisherman.test.ts
|
|
1064
|
+
git commit -m "feat: add Fisherman.lookupData for read-only data questions"
|
|
1065
|
+
```
|
|
1066
|
+
|
|
1067
|
+
---
|
|
1068
|
+
|
|
1069
|
+
### Task 5: Pilot's queryApi tool
|
|
1070
|
+
|
|
1071
|
+
**Files:**
|
|
1072
|
+
- Modify: `src/ai/pilot.ts:690` (the tool spread in `sendToPilot`), `src/ai/pilot.ts:736-779` (`buildPreconditionTool`), `src/ai/pilot.ts:454-465` (the planTest decision block), `src/ai/pilot.ts:1150` (the empty-dropdown diagnostic), `src/ai/pilot.ts:1167-1178` (the Pilot-only tool block in `getSystemPrompt`)
|
|
1073
|
+
- Modify: `src/ai/rules.ts:165-178` (`dataProtectionRules`)
|
|
1074
|
+
- Test: Create `tests/integration/pilot-query-api.test.ts`
|
|
1075
|
+
|
|
1076
|
+
**Interfaces:**
|
|
1077
|
+
- Consumes: `Fisherman.lookupData(question, scopeUrl, sessionName)` from Task 4; `Test.addNote`, `Test.startUrl`, `Test.sessionName`.
|
|
1078
|
+
- Produces: `Pilot.buildFishermanTools(task)` (private, renamed from `buildPreconditionTool`) returning `{ precondition, queryApi }`. `queryApi` takes `{ question: string }` and returns `{ answered: true, answer }` or `{ answered: false, reason }`. No other module calls it — `sendToPilot` already spreads the builder's result, so the tool reaches planning, new-page review and progress analysis with no further wiring.
|
|
1079
|
+
|
|
1080
|
+
- [ ] **Step 1: Write the failing tests**
|
|
1081
|
+
|
|
1082
|
+
Create `tests/integration/pilot-query-api.test.ts`:
|
|
1083
|
+
|
|
1084
|
+
```ts
|
|
1085
|
+
import { createOpenAI } from '@ai-sdk/openai';
|
|
1086
|
+
import { LLMock } from '@copilotkit/aimock';
|
|
1087
|
+
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test';
|
|
1088
|
+
import { ActionResult } from '../../src/action-result.ts';
|
|
1089
|
+
import { Pilot } from '../../src/ai/pilot.ts';
|
|
1090
|
+
import { Provider } from '../../src/ai/provider.ts';
|
|
1091
|
+
import { ConfigParser } from '../../src/config.ts';
|
|
1092
|
+
import { Test } from '../../src/test-plan.ts';
|
|
1093
|
+
|
|
1094
|
+
function toolCall(id: string, name: string, args: Record<string, any>) {
|
|
1095
|
+
return { id, name, arguments: JSON.stringify(args) };
|
|
1096
|
+
}
|
|
1097
|
+
|
|
1098
|
+
describe('Pilot queryApi', () => {
|
|
1099
|
+
let mock: LLMock;
|
|
1100
|
+
let provider: Provider;
|
|
1101
|
+
let lookupCalls: Array<{ question: string; scopeUrl?: string }>;
|
|
1102
|
+
|
|
1103
|
+
beforeAll(async () => {
|
|
1104
|
+
mock = new LLMock({ port: 0, logLevel: 'silent' });
|
|
1105
|
+
await mock.start();
|
|
1106
|
+
|
|
1107
|
+
const openai = createOpenAI({ baseURL: `${mock.url}/v1`, apiKey: 'test-key', compatibility: 'compatible' });
|
|
1108
|
+
ConfigParser.setupTestConfig();
|
|
1109
|
+
provider = new Provider({ model: openai.chat('test-model'), config: {} });
|
|
1110
|
+
});
|
|
1111
|
+
|
|
1112
|
+
beforeEach(() => {
|
|
1113
|
+
mock.clearRequests();
|
|
1114
|
+
mock.resetMatchCounts();
|
|
1115
|
+
mock.clearFixtures();
|
|
1116
|
+
lookupCalls = [];
|
|
1117
|
+
});
|
|
1118
|
+
|
|
1119
|
+
afterAll(async () => {
|
|
1120
|
+
await mock.stop();
|
|
1121
|
+
});
|
|
1122
|
+
|
|
1123
|
+
function createPilot(fisherman?: any): Pilot {
|
|
1124
|
+
const deps = {
|
|
1125
|
+
ai: provider,
|
|
1126
|
+
config: ConfigParser.getInstance().getConfig(),
|
|
1127
|
+
explorer: {},
|
|
1128
|
+
stateManager: { getCurrentState: () => null, otherTabs: [] },
|
|
1129
|
+
requestStore: { getFailedRequests: () => [] },
|
|
1130
|
+
playwrightRecorder: {},
|
|
1131
|
+
};
|
|
1132
|
+
const researcher = { summary: async () => 'A list of tests' };
|
|
1133
|
+
const pilot = new Pilot(deps as any, {} as any, researcher as any);
|
|
1134
|
+
if (fisherman) pilot.setFisherman(fisherman);
|
|
1135
|
+
return pilot;
|
|
1136
|
+
}
|
|
1137
|
+
|
|
1138
|
+
function availableFisherman(summary: string, success = true) {
|
|
1139
|
+
return {
|
|
1140
|
+
isAvailable: () => true,
|
|
1141
|
+
lookupData: async (question: string, scopeUrl?: string) => {
|
|
1142
|
+
lookupCalls.push({ question, scopeUrl });
|
|
1143
|
+
return { success, summary, created: [], failed: [] };
|
|
1144
|
+
},
|
|
1145
|
+
};
|
|
1146
|
+
}
|
|
1147
|
+
|
|
1148
|
+
function planningTask(): { task: Test; state: ActionResult } {
|
|
1149
|
+
const task = new Test('filter the list by label', 'normal', ['the list narrows'], '/projects/alpha-shop/tests');
|
|
1150
|
+
const state = new ActionResult({ url: '/projects/alpha-shop/tests', title: 'Tests', h1: 'Tests' });
|
|
1151
|
+
return { task, state };
|
|
1152
|
+
}
|
|
1153
|
+
|
|
1154
|
+
it('offers queryApi while planning', async () => {
|
|
1155
|
+
const { task, state } = planningTask();
|
|
1156
|
+
mock.on({}, { content: 'PROGRESS: ready\nNEXT: open the filter' });
|
|
1157
|
+
|
|
1158
|
+
await createPilot(availableFisherman('unused')).planTest(task, state);
|
|
1159
|
+
|
|
1160
|
+
expect(JSON.stringify(mock.getRequests()[0]?.body?.tools)).toContain('queryApi');
|
|
1161
|
+
});
|
|
1162
|
+
|
|
1163
|
+
it('answers from Fisherman and records the answer as a note, not a step', async () => {
|
|
1164
|
+
const { task, state } = planningTask();
|
|
1165
|
+
mock.on({ sequenceIndex: 0 }, { toolCalls: [toolCall('q1', 'queryApi', { question: 'which labels exist?' })] });
|
|
1166
|
+
mock.on({}, { content: 'PROGRESS: labels are available\nNEXT: open the label filter' });
|
|
1167
|
+
|
|
1168
|
+
const plan = await createPilot(availableFisherman('Two labels exist: Bug and Urgent')).planTest(task, state);
|
|
1169
|
+
|
|
1170
|
+
expect(lookupCalls).toEqual([{ question: 'which labels exist?', scopeUrl: '/projects/alpha-shop/tests' }]);
|
|
1171
|
+
expect(plan).toContain('open the label filter');
|
|
1172
|
+
expect(Object.values(task.notes).some((n: any) => n.message.includes('Two labels exist: Bug and Urgent'))).toBe(true);
|
|
1173
|
+
expect(Object.keys(task.steps)).toHaveLength(0);
|
|
1174
|
+
});
|
|
1175
|
+
|
|
1176
|
+
it('tells the model to fall back to the page when there is no API access', async () => {
|
|
1177
|
+
const { task, state } = planningTask();
|
|
1178
|
+
mock.on({ sequenceIndex: 0 }, { toolCalls: [toolCall('q1', 'queryApi', { question: 'which labels exist?' })] });
|
|
1179
|
+
mock.on({}, { content: 'PROGRESS: no API\nNEXT: read the labels from the page' });
|
|
1180
|
+
|
|
1181
|
+
await createPilot().planTest(task, state);
|
|
1182
|
+
|
|
1183
|
+
expect(JSON.stringify(mock.getRequests()[1]?.body)).toContain('No API access is configured');
|
|
1184
|
+
});
|
|
1185
|
+
|
|
1186
|
+
it('passes the failure reason through when the lookup cannot answer', async () => {
|
|
1187
|
+
const { task, state } = planningTask();
|
|
1188
|
+
mock.on({ sequenceIndex: 0 }, { toolCalls: [toolCall('q1', 'queryApi', { question: 'which labels exist?' })] });
|
|
1189
|
+
mock.on({}, { content: 'PROGRESS: unanswered\nNEXT: read the labels from the page' });
|
|
1190
|
+
|
|
1191
|
+
await createPilot(availableFisherman('No read endpoints are known for this scope', false)).planTest(task, state);
|
|
1192
|
+
|
|
1193
|
+
expect(JSON.stringify(mock.getRequests()[1]?.body)).toContain('No read endpoints are known for this scope');
|
|
1194
|
+
expect(Object.values(task.notes)).toHaveLength(0);
|
|
1195
|
+
});
|
|
1196
|
+
});
|
|
1197
|
+
```
|
|
1198
|
+
|
|
1199
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
1200
|
+
|
|
1201
|
+
```bash
|
|
1202
|
+
bun test tests/integration/pilot-query-api.test.ts
|
|
1203
|
+
```
|
|
1204
|
+
|
|
1205
|
+
Expected: FAIL — no `queryApi` tool is offered, so the mocked tool call is never executed.
|
|
1206
|
+
|
|
1207
|
+
- [ ] **Step 3: Rename the builder and add the tool**
|
|
1208
|
+
|
|
1209
|
+
In `src/ai/pilot.ts`, rename `buildPreconditionTool` to `buildFishermanTools` and update its single call site in `sendToPilot`:
|
|
1210
|
+
|
|
1211
|
+
```ts
|
|
1212
|
+
const tools = { ...this.pickPlanningTools(), ...this.buildFishermanTools(opts.task) };
|
|
1213
|
+
```
|
|
1214
|
+
|
|
1215
|
+
Inside the builder, add `queryApi` alongside the existing `precondition` in the returned object:
|
|
1216
|
+
|
|
1217
|
+
```ts
|
|
1218
|
+
queryApi: tool({
|
|
1219
|
+
description: dedent`
|
|
1220
|
+
Read the API to learn what data already exists, changing nothing.
|
|
1221
|
+
Ask a question about existing records: which ones are there, what they are called, whether a particular one exists.
|
|
1222
|
+
Use it before precondition() to see whether suitable data is already available, and whenever a step needs the exact name or id of a record that is already there.
|
|
1223
|
+
It never creates, edits or deletes anything — precondition() does that.
|
|
1224
|
+
`,
|
|
1225
|
+
inputSchema: z.object({
|
|
1226
|
+
question: z.string().describe('What to find out about data that already exists'),
|
|
1227
|
+
}),
|
|
1228
|
+
execute: async ({ question }) => {
|
|
1229
|
+
tag('info').log(`Query API: ${question}`);
|
|
1230
|
+
debugLog(`queryApi: ${question}, fisherman: ${this.fisherman?.isAvailable() ? 'available' : 'none'}`);
|
|
1231
|
+
|
|
1232
|
+
if (!this.fisherman || !this.fisherman.isAvailable()) {
|
|
1233
|
+
return { answered: false, reason: 'No API access is configured, so existing data cannot be queried. Judge from the page instead.' };
|
|
1234
|
+
}
|
|
1235
|
+
|
|
1236
|
+
const result = await this.fisherman.lookupData(question, task.startUrl, task.sessionName);
|
|
1237
|
+
|
|
1238
|
+
if (!result.success) {
|
|
1239
|
+
tag('warning').log(`Query API unanswered: ${result.summary}`);
|
|
1240
|
+
return { answered: false, reason: result.summary || 'The API could not answer this question' };
|
|
1241
|
+
}
|
|
1242
|
+
|
|
1243
|
+
task.addNote(`Queried API: ${question} — ${result.summary}`);
|
|
1244
|
+
tag('success').log(`Query API: ${result.summary}`);
|
|
1245
|
+
return { answered: true, answer: result.summary };
|
|
1246
|
+
},
|
|
1247
|
+
}),
|
|
1248
|
+
```
|
|
1249
|
+
|
|
1250
|
+
- [ ] **Step 4: Update the Pilot system prompt**
|
|
1251
|
+
|
|
1252
|
+
In `getSystemPrompt`, replace the block that currently opens `YOUR Pilot-only tool: precondition(description) — create FRESH disposable test data via API. Never request users. Use when:` with:
|
|
1253
|
+
|
|
1254
|
+
```ts
|
|
1255
|
+
YOUR Pilot-only tools, both over the API:
|
|
1256
|
+
|
|
1257
|
+
queryApi(question) — read what data already exists. It changes nothing. Use it to check whether
|
|
1258
|
+
suitable data is already there before creating any, and to get the exact name or id of an existing
|
|
1259
|
+
record a step must act on.
|
|
1260
|
+
|
|
1261
|
+
precondition(description) — create FRESH disposable test data. Never request users. Use when:
|
|
1262
|
+
|
|
1263
|
+
- Scenario edits/deletes/modifies an item → create a disposable target ("1 post").
|
|
1264
|
+
- Scenario needs auxiliary data (labels, categories, statuses for filtering).
|
|
1265
|
+
- Tester failed because required data is missing (empty dropdown, empty list).
|
|
1266
|
+
```
|
|
1267
|
+
|
|
1268
|
+
Leave the `Skip precondition() when:` list and the `Describe WHAT to create` paragraph exactly as they are.
|
|
1269
|
+
|
|
1270
|
+
In the diagnostic-patterns list, replace the empty-dropdown line with:
|
|
1271
|
+
|
|
1272
|
+
```ts
|
|
1273
|
+
- Empty dropdown/list when items expected → queryApi() to confirm none exist, then precondition() to create them.
|
|
1274
|
+
```
|
|
1275
|
+
|
|
1276
|
+
- [ ] **Step 5: Update the planTest instruction block**
|
|
1277
|
+
|
|
1278
|
+
In `planTest`, change the first line of the decision block from `FIRST: Decide if precondition() is needed.` to:
|
|
1279
|
+
|
|
1280
|
+
```ts
|
|
1281
|
+
FIRST: Decide if precondition() is needed. When the page does not settle whether suitable data
|
|
1282
|
+
already exists, call queryApi() to find out before creating any.
|
|
1283
|
+
```
|
|
1284
|
+
|
|
1285
|
+
- [ ] **Step 6: Allow reads under a no-mutation constraint**
|
|
1286
|
+
|
|
1287
|
+
In `src/ai/rules.ts`, inside `dataProtectionRules`, add one sentence after the paragraph beginning `Do not use Fisherman or API data preparation to bypass a no-mutation`:
|
|
1288
|
+
|
|
1289
|
+
```ts
|
|
1290
|
+
Reading through the API to establish what already exists is not a mutation and stays allowed
|
|
1291
|
+
under a read-only constraint.
|
|
1292
|
+
```
|
|
1293
|
+
|
|
1294
|
+
- [ ] **Step 7: Run the tests to verify they pass**
|
|
1295
|
+
|
|
1296
|
+
```bash
|
|
1297
|
+
bun run format
|
|
1298
|
+
bun test tests/integration/
|
|
1299
|
+
```
|
|
1300
|
+
|
|
1301
|
+
Expected: PASS across the whole integration suite. `pilot-expectations.test.ts` and `planner.test.ts` must stay green — the prompt edits touch text they may assert on.
|
|
1302
|
+
|
|
1303
|
+
- [ ] **Step 8: Commit**
|
|
1304
|
+
|
|
1305
|
+
```bash
|
|
1306
|
+
git add src/ai/pilot.ts src/ai/rules.ts tests/integration/pilot-query-api.test.ts
|
|
1307
|
+
git commit -m "feat: let Pilot query existing data through queryApi"
|
|
1308
|
+
```
|
|
1309
|
+
|
|
1310
|
+
---
|
|
1311
|
+
|
|
1312
|
+
### Task 6: Documentation and changelog
|
|
1313
|
+
|
|
1314
|
+
**Files:**
|
|
1315
|
+
- Modify: `docs/reference/configuration.md` (the Fisherman section)
|
|
1316
|
+
- Modify: `CHANGELOG.md`
|
|
1317
|
+
|
|
1318
|
+
**Interfaces:**
|
|
1319
|
+
- Consumes: everything built in Tasks 1–5. Produces no code.
|
|
1320
|
+
|
|
1321
|
+
- [ ] **Step 1: Run the full suite and the linter**
|
|
1322
|
+
|
|
1323
|
+
```bash
|
|
1324
|
+
bun run format
|
|
1325
|
+
bun run lint:fix
|
|
1326
|
+
bun test
|
|
1327
|
+
```
|
|
1328
|
+
|
|
1329
|
+
Expected: PASS. Fix anything the linter rewrites before continuing.
|
|
1330
|
+
|
|
1331
|
+
- [ ] **Step 2: Document the read capability**
|
|
1332
|
+
|
|
1333
|
+
In `docs/reference/configuration.md`, find the Fisherman section and add a short subsection describing that Fisherman now also answers questions about existing data over GET, that Pilot reaches it through `queryApi`, and that in replicate mode the read endpoints come from successful GET XHRs observed in the browser (recorded as endpoint + query-parameter names only, with no response body). Keep it to a paragraph — no example that names a specific site or endpoint.
|
|
1334
|
+
|
|
1335
|
+
- [ ] **Step 3: Write the changelog entry**
|
|
1336
|
+
|
|
1337
|
+
Invoke the `/changelog` skill to generate the CHANGELOG.md entry from the commits on this branch. Do not hand-write it.
|
|
1338
|
+
|
|
1339
|
+
- [ ] **Step 4: Commit**
|
|
1340
|
+
|
|
1341
|
+
```bash
|
|
1342
|
+
git add docs/reference/configuration.md CHANGELOG.md
|
|
1343
|
+
git commit -m "docs: describe Fisherman read lookups and queryApi"
|
|
1344
|
+
```
|
|
1345
|
+
|
|
1346
|
+
- [ ] **Step 5: Report back**
|
|
1347
|
+
|
|
1348
|
+
Summarise for the user: what `queryApi` does, that GET endpoints are now recorded, and that regression coverage has not been run. **Do not add the `regression` label or dispatch the regression workflow** — only the user decides that.
|
|
1349
|
+
|
|
1350
|
+
---
|
|
1351
|
+
|
|
1352
|
+
## Self-Review
|
|
1353
|
+
|
|
1354
|
+
**Spec coverage** — every design decision maps to a task: GET capture without bodies → Task 1; read/write endpoint families → Task 2; schema-enforced read-only plus body preview and prose `finish` → Task 3; `lookupData`, the shared loop, and the removed permanent `disabled` mutation → Task 4; the `queryApi` tool, prompts and rule → Task 5; docs → Task 6.
|
|
1355
|
+
|
|
1356
|
+
**Type consistency** — `EndpointFamily` is declared once (`src/api/request-store.ts`, Task 2) and consumed by `toEndpointList`, `getRequestsForScope` and `Fisherman.buildEndpointList`. `readOnly` is the flag name in `createFishermanTools` throughout. `lookupData(question, scopeUrl, sessionName)` has the same argument order in Task 4's definition and Task 5's call. `addReadRequest` is the store method name in Tasks 1, 2 and 4's test helper.
|
|
1357
|
+
|
|
1358
|
+
**Two consequences checked against the code before this plan was committed:**
|
|
1359
|
+
|
|
1360
|
+
- `detectMode` flips to `replicate` on any captured request, and read captures now count — so `isAvailable()` becomes true on a GET-only store where it used to be false. The one consumer that reads the write list off the back of it, `src/ai/planner.ts:434`, already guards with `if (endpointList)`, so an empty write list simply skips the `<api_data_preparation>` block. `prepareData` is likewise unaffected: its list stays write-only and it returns an honest failure when empty (Task 4, Step 3). If a future change makes `isAvailable()` mean "can write", it must be split per family.
|
|
1361
|
+
- `Queried API:` is a new note prefix, and CLAUDE.md keeps noise prefixes a closed set owned by `src/utils/test-plan-markdown.ts`. That set is `NOISE_PREFIXES = ['Test started', 'Finish requested:', 'Session name:']` plus special routing for `Pilot:`. `Precondition:` is not registered either — both are plain prose that flows through unchanged. Nothing to register.
|