opensteer 0.8.9 → 0.8.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,22 +1,10 @@
1
- # Opensteer Request Workflow
1
+ # Request Plan Pipeline
2
2
 
3
3
  If you haven't decided whether this workflow applies, see the task triage in SKILL.md.
4
4
 
5
- Use this workflow when the deliverable is a custom API, a replayable request plan, or a lower-overhead path than full browser automation.
5
+ ## The Deliverable
6
6
 
7
- ## Sections
8
-
9
- - [Critical Rules](#critical-rules)
10
- - [Standard Loop](#standard-loop)
11
- - [Transport Selection](#transport-selection)
12
- - [SDK Flow](#sdk-flow)
13
- - [CLI Equivalents](#cli-equivalents)
14
- - [Transport Probing](#transport-probing)
15
- - [Auth Token Acquisition](#auth-token-acquisition)
16
- - [Input Formats](#input-formats)
17
- - [Practical Guidance](#practical-guidance)
18
- - [When To Use Request Capture vs DOM Extraction](#when-to-use-request-capture-vs-dom-extraction)
19
- - [Error Recovery](#error-recovery)
7
+ The deliverable is a **persisted request plan** that works via `request.execute`. `rawRequest()` is a diagnostic probe — its output is never the final answer. You are not done until `request.execute` returns valid data from a stored plan.
20
8
 
21
9
  ## Critical Rules
22
10
 
@@ -26,19 +14,6 @@ Use this workflow when the deliverable is a custom API, a replayable request pla
26
14
  4. `clearNetwork()` permanently removes records with tombstoning. Cleared records cannot be resurrected by late-arriving browser events.
27
15
  5. `waitForNetwork()` watches for NEW records only. It snapshots existing records and polls for ones that were not present at the start. It does NOT return historical matches.
28
16
 
29
- ## Standard Loop
30
-
31
- 1. Trigger the real browser action that causes the request inside a stable workspace.
32
- 2. Name the important navigation or interaction with `captureNetwork` on the action itself.
33
- 3. Inspect the captured traffic with `queryNetwork()` and isolate the relevant records.
34
- 4. Use `tagNetwork()` to label interesting records for later lookup if you want extra manual labels.
35
- 5. Probe the request with `rawRequest()` — try `direct-http` first, then `context-http`.
36
- 6. Infer a request plan from the probed record — pass `transport` if you proved portability.
37
- 7. Add recipes or auth recipes if replay needs deterministic setup.
38
- 8. Replay the plan from code — works immediately, no extra steps.
39
-
40
- This workflow should carry equal weight with DOM automation. Use it whenever the browser page is only the launcher for the real target request.
41
-
42
17
  ## Transport Selection
43
18
 
44
19
  - `direct-http`: the request is replayable without a browser.
@@ -48,234 +23,378 @@ This workflow should carry equal weight with DOM automation. Use it whenever the
48
23
 
49
24
  When in doubt, start with browser-backed capture first. Opensteer treats browser-backed replay as a first-class path, not a fallback.
50
25
 
51
- ## SDK Flow
26
+ ## Pipeline Phases
52
27
 
53
- 1. Start a workspace-backed browser flow and tag navigation.
28
+ Work through each phase in order. Do NOT skip phases. Each phase has exit criteria — verify them before proceeding.
54
29
 
55
- ```ts
56
- await opensteer.open();
57
- await opensteer.goto({
58
- url: "https://example.com/app",
59
- captureNetwork: "page-load",
60
- });
30
+ ### Phase 1: Capture
61
31
 
62
- await opensteer.click({
63
- description: "load products",
64
- captureNetwork: "products-load",
65
- });
32
+ Trigger the real browser action that causes the request. Name the capture.
33
+
34
+ ```bash
35
+ opensteer open https://example.com/app --workspace demo
36
+ opensteer run page.goto --workspace demo \
37
+ --input-json '{"url":"https://example.com/app","captureNetwork":"page-load"}'
66
38
  ```
67
39
 
68
- 2. Inspect the captured traffic.
40
+ For interactions that trigger API calls (search, filter, load-more):
69
41
 
70
- ```ts
71
- const records = await opensteer.queryNetwork({
72
- capture: "products-load",
73
- includeBodies: true,
74
- limit: 20,
75
- });
42
+ ```bash
43
+ opensteer run dom.click --workspace demo \
44
+ --input-json '{"target":{"kind":"description","description":"load products"},"captureNetwork":"products-load"}'
76
45
  ```
77
46
 
78
- 3. Probe the request try `direct-http` first to test portability.
79
-
80
- ```ts
81
- const response = await opensteer.rawRequest({
82
- transport: "direct-http",
83
- url: "https://example.com/api/products",
84
- method: "POST",
85
- body: {
86
- json: { page: 1 },
87
- },
88
- });
89
- // If direct-http returns 200, the API is portable (no browser needed).
90
- // If it fails, try context-http — the API needs browser session state.
47
+ **EXIT CRITERIA:** You have at least one named capture. If `queryNetwork` returns empty after capture, see Error Recovery.
48
+
49
+ ### Phase 2: Discover
50
+
51
+ Query captured traffic to isolate the API calls worth replaying. Ignore static assets, analytics, and third-party scripts.
52
+
53
+ ```bash
54
+ opensteer run network.query --workspace demo \
55
+ --input-json '{"capture":"products-load","includeBodies":true,"limit":20}'
91
56
  ```
92
57
 
93
- 4. Infer a request planpass `transport` if you proved portability.
58
+ Examine the results. Look for first-party JSON APIs requests returning `application/json` with data relevant to the task.
94
59
 
95
- ```ts
96
- await opensteer.inferRequestPlan({
97
- recordId: response.recordId,
98
- key: "products.search",
99
- version: "v1",
100
- transport: "direct-http", // use the transport you proved works
101
- });
60
+ If the first query is too broad, filter by hostname, path, or method:
61
+
62
+ ```bash
63
+ opensteer run network.query --workspace demo \
64
+ --input-json '{"capture":"products-load","hostname":"api.example.com","method":"GET","includeBodies":true,"limit":10}'
102
65
  ```
103
66
 
104
- 5. Tag additional records if you want extra manual labels on top of the action capture name.
67
+ **EXIT CRITERIA:** You have identified at least one candidate API URL with its method, recordId, and response shape.
68
+
69
+ ### Phase 3: Probe (Diagnostic Only)
105
70
 
106
- ```ts
107
- await opensteer.tagNetwork({
108
- tag: "products-load",
109
- });
71
+ `rawRequest()` is a diagnostic tool. Use it to determine:
72
+ 1. Which transport works (`direct-http` vs `context-http`)
73
+ 2. Whether the API returns the expected data shape
74
+ 3. Whether auth headers are actually required
75
+
76
+ `rawRequest()` output is NOT the deliverable. Do NOT return rawRequest results to the user as the final answer. Always proceed to Phase 4.
77
+
78
+ Test portability — try `direct-http` first:
79
+
80
+ ```bash
81
+ opensteer run request.raw --workspace demo \
82
+ --input-json '{"transport":"direct-http","url":"https://api.example.com/products","method":"GET"}'
110
83
  ```
111
84
 
112
- Network history is SQLite-backed. Records are written to SQLite only for actions that opt into `captureNetwork`, or when later persisted by network inspection flows. The database initializes on first use.
85
+ If `direct-http` returns 200, the API is portable. If it fails (403/401), try `context-http`:
86
+
87
+ ```bash
88
+ opensteer run request.raw --workspace demo \
89
+ --input-json '{"transport":"context-http","url":"https://api.example.com/products","method":"GET"}'
90
+ ```
91
+
92
+ **EXIT CRITERIA:** You know which transport works and have a successful response. Note the `recordId` from the probe response — you will use it in Phase 4.
93
+
94
+ ### Phase 4: Infer Plan
113
95
 
114
- 6. Replay the plan from code.
96
+ Create a request plan from the probed record. Pass the `transport` you proved works.
115
97
 
116
- ```ts
117
- const result = await opensteer.request("products.search", {
118
- query: { q: "laptop" },
119
- });
98
+ ```bash
99
+ opensteer run request-plan.infer --workspace demo \
100
+ --input-json '{"recordId":"<recordId-from-phase-3>","key":"products.search","version":"v1","transport":"direct-http"}'
120
101
  ```
121
102
 
122
- 7. Add a recipe or auth recipe if replay needs deterministic setup.
103
+ If you proved `direct-http` works, always pass `transport: "direct-http"` so the plan is portable.
104
+
105
+ If `inferRequestPlan` throws "registry record already exists", bump the version (e.g., `v2`).
106
+
107
+ **EXIT CRITERIA:** Plan is persisted. You can verify with `request-plan.get`.
123
108
 
124
- ```ts
125
- await opensteer.runRecipe({
126
- key: "products.setup",
127
- });
109
+ ### Phase 5: Validate Auth Classification
128
110
 
129
- await opensteer.runAuthRecipe({
130
- key: "products.auth",
131
- });
111
+ `inferRequestPlan` records auth metadata by observing headers on the captured request. This is **often wrong**. If the browser sent an `Authorization` header, the plan records `auth.strategy: "bearer-token"` even if the API works without auth.
112
+
113
+ **MANDATORY VALIDATION:**
114
+
115
+ 1. Read the inferred plan:
116
+
117
+ ```bash
118
+ opensteer run request-plan.get --workspace demo \
119
+ --input-json '{"key":"products.search","version":"v1"}'
132
120
  ```
133
121
 
134
- ## CLI Equivalents
122
+ 2. Check the `auth` field in `payload`. If `auth` is absent or `undefined`, auth is not detected — skip to Phase 6.
123
+
124
+ 3. If `auth.strategy` is set, test whether the API actually needs it. Run a raw request to the same URL with NO auth headers:
135
125
 
136
126
  ```bash
137
- opensteer open https://example.com/app --workspace demo
138
- opensteer run page.goto --workspace demo \
139
- --input-json '{"url":"https://example.com/app","captureNetwork":"page-load"}'
140
- opensteer click --workspace demo --description "load products"
141
- # or with captureNetwork: opensteer run dom.click --workspace demo \
142
- # --input-json '{"target":{"kind":"description","description":"load products"},"captureNetwork":"products-load"}'
143
- opensteer run network.query --workspace demo \
144
- --input-json '{"capture":"products-load","includeBodies":true,"limit":20}'
145
127
  opensteer run request.raw --workspace demo \
146
- --input-json '{"transport":"direct-http","url":"https://example.com/api/products","method":"POST","body":{"json":{"page":1}}}'
147
- opensteer run request-plan.infer --workspace demo \
148
- --input-json '{"recordId":"rec_123","key":"products.search","version":"v1","transport":"direct-http"}'
149
- opensteer run request.execute --workspace demo \
150
- --input-json '{"key":"products.search","query":{"q":"laptop"}}'
128
+ --input-json '{"transport":"direct-http","url":"<the-api-url>","method":"GET"}'
151
129
  ```
152
130
 
153
- ## Transport Probing
131
+ 4. If it returns 200 without auth headers, auth is **spurious** — the browser attached a token the API doesn't enforce. Rewrite the plan with corrected auth:
154
132
 
155
- Test each discovered API with multiple transports to determine portability:
133
+ ```bash
134
+ opensteer run request-plan.write --workspace demo \
135
+ --input-json '{
136
+ "key":"products.search",
137
+ "version":"v1",
138
+ "tags":["products","search"],
139
+ "provenance":{"source":"manual","notes":"Auth removed — API is public, bearer token was incidental."},
140
+ "payload":{
141
+ ...existing payload with auth field removed...
142
+ }
143
+ }'
144
+ ```
145
+
146
+ Copy the full existing `payload` from `request-plan.get`, remove or null out the `auth` field, and write it back.
147
+
148
+ 5. If the no-auth probe returns 401/403, auth IS required. Keep the auth classification and proceed. You will create an auth recipe in Phase 8 after testing the plan.
149
+
150
+ **EXIT CRITERIA:** The plan's `auth` field accurately reflects whether auth is required.
151
+
152
+ ### Phase 6: Annotate Parameters
153
+
154
+ `inferRequestPlan` dumps all query and body parameters into `defaultQuery`/`defaultBody` without distinguishing variable from fixed.
155
+
156
+ 1. Read the plan with `request-plan.get`.
156
157
 
157
- ```ts
158
- const direct = await opensteer.rawRequest({
159
- transport: "direct-http",
160
- url: discoveredUrl,
161
- method: "GET",
162
- });
158
+ 2. Examine each parameter in `defaultQuery`:
159
+ - **Variable:** values that change per invocation — search terms, page numbers, offsets, dates, user-specific IDs
160
+ - **Fixed:** values constant for this API — site keys, platform identifiers, API versions, channel strings
163
161
 
164
- const context = await opensteer.rawRequest({
165
- transport: "context-http",
166
- url: discoveredUrl,
167
- method: "GET",
168
- });
162
+ 3. Rewrite the plan with the `parameters` field annotating variable inputs:
163
+
164
+ ```bash
165
+ opensteer run request-plan.write --workspace demo \
166
+ --input-json '{
167
+ "key":"products.search",
168
+ "version":"v1",
169
+ "payload":{
170
+ ...existing payload...,
171
+ "parameters":[
172
+ {"name":"keyword","in":"query","required":true,"description":"Search term"},
173
+ {"name":"count","in":"query","defaultValue":"24","description":"Results per page"},
174
+ {"name":"offset","in":"query","defaultValue":"0","description":"Pagination offset"}
175
+ ]
176
+ }
177
+ }'
169
178
  ```
170
179
 
171
- If `direct-http` returns 200, the API is portable and does not need a browser for future calls. If only `context-http` works, the API depends on browser session state.
180
+ Variable params remain in `defaultQuery` as initial values. The `parameters` field documents which ones a caller should override via `request.execute` input.
181
+
182
+ **EXIT CRITERIA:** The plan's `parameters` field lists all variable inputs with descriptions.
183
+
184
+ ### Phase 7: Test Plan
172
185
 
173
- After proving portability, infer the plan with an explicit transport override:
186
+ Execute the plan through `request.execute`, NOT `rawRequest`:
174
187
 
175
- ```ts
176
- await opensteer.inferRequestPlan({
177
- recordId: records.records[0]!.id,
178
- key: "products.search.portable",
179
- version: "v1",
180
- transport: "direct-http",
181
- });
188
+ ```bash
189
+ opensteer run request.execute --workspace demo \
190
+ --input-json '{"key":"products.search","version":"v1","query":{"keyword":"laptop","count":"10"}}'
182
191
  ```
183
192
 
193
+ **GATE:**
194
+ - If `request.execute` returns valid data → proceed to Phase 9 (Done).
195
+ - If `request.execute` returns 401/403 and Phase 5 confirmed auth is required → proceed to Phase 8 (Auth Recipe).
196
+ - If `request.execute` fails with another error → see Error Recovery.
197
+
198
+ ### Phase 8: Auth Recipe (Conditional)
199
+
200
+ Enter this phase ONLY if Phase 5 confirmed auth is genuinely required AND Phase 7 failed with 401/403.
201
+
202
+ #### Step 8a: Discover Auth Endpoint
203
+
204
+ Search captured traffic for OAuth, token, or login endpoints:
205
+
184
206
  ```bash
185
- opensteer run request-plan.infer --workspace demo \
186
- --input-json '{"recordId":"rec_123","key":"products.search.portable","version":"v1","transport":"direct-http"}'
207
+ opensteer run network.query --workspace demo \
208
+ --input-json '{"path":"/oauth","includeBodies":true,"limit":10}'
209
+ opensteer run network.query --workspace demo \
210
+ --input-json '{"path":"/token","includeBodies":true,"limit":10}'
211
+ opensteer run network.query --workspace demo \
212
+ --input-json '{"path":"/auth","includeBodies":true,"limit":10}'
187
213
  ```
188
214
 
189
- ## Auth Token Acquisition
190
-
191
- When you discover an auth endpoint, acquire a token and use it to probe for data APIs that may be behind auth:
192
-
193
- ```ts
194
- const tokenResp = await opensteer.rawRequest({
195
- transport: "direct-http",
196
- url: "https://example.com/api/oauth/token?scope=guest",
197
- method: "POST",
198
- });
199
-
200
- let parsed = tokenResp.data;
201
- if (parsed === undefined) {
202
- const body = tokenResp.response.body;
203
- if (!body) {
204
- throw new Error("Token response had no body");
205
- }
206
- parsed = JSON.parse(Buffer.from(body.data, "base64").toString("utf8"));
207
- }
208
-
209
- const token = String((parsed as { access_token: unknown }).access_token);
210
-
211
- const authed = await opensteer.rawRequest({
212
- transport: "direct-http",
213
- url: "https://example.com/api/products",
214
- method: "GET",
215
- headers: [{ name: "Authorization", value: `Bearer ${token}` }],
216
- });
215
+ Examine responses to find the endpoint that returns an access token.
216
+
217
+ #### Step 8b: Probe Auth Endpoint
218
+
219
+ Test the auth endpoint with `request.raw`:
220
+
221
+ ```bash
222
+ opensteer run request.raw --workspace demo \
223
+ --input-json '{
224
+ "transport":"direct-http",
225
+ "url":"https://example.com/api/oauth/token",
226
+ "method":"POST",
227
+ "body":{"json":{"grant_type":"client_credentials"}}
228
+ }'
217
229
  ```
218
230
 
219
- ## Input Formats
231
+ Verify it returns a token. Note the response shape (e.g., `{ "access_token": "..." }`).
220
232
 
221
- `rawRequest` expects specific shapes:
233
+ #### Step 8c: Create Auth Recipe
222
234
 
223
- - `headers`: array of `[{ name, value }]`, not `{ key: value }`.
224
- - `body`: one of `{ json: { ... } }`, `{ text: "..." }`, or `{ base64: "..." }`. Not raw strings.
225
- - `request.execute` semantic input includes `key` inside the JSON object. The SDK convenience wrapper `opensteer.request(key, input)` adds that for you.
235
+ Write an auth recipe that acquires a fresh token and maps it to request headers:
226
236
 
227
- ## Practical Guidance
237
+ ```bash
238
+ opensteer run auth-recipe.write --workspace demo \
239
+ --input-json '{
240
+ "key":"example.auth",
241
+ "version":"v1",
242
+ "payload":{
243
+ "description":"Acquire bearer token for example.com API",
244
+ "steps":[
245
+ {
246
+ "kind":"directRequest",
247
+ "request":{
248
+ "url":"https://example.com/api/oauth/token",
249
+ "transport":"direct-http",
250
+ "method":"POST",
251
+ "body":{"json":{"grant_type":"client_credentials"}}
252
+ },
253
+ "capture":{
254
+ "bodyJsonPointer":{"pointer":"/access_token","saveAs":"token"}
255
+ }
256
+ }
257
+ ],
258
+ "outputs":{
259
+ "headers":{"Authorization":"Bearer {{token}}"}
260
+ }
261
+ }
262
+ }'
263
+ ```
228
264
 
229
- Mandatory steps:
265
+ **Recipe step types you can use:**
266
+ - `directRequest` — HTTP request outside the browser (portable, no session needed)
267
+ - `sessionRequest` — HTTP request using browser session state (cookies, etc.)
268
+ - `request` — generic request step
269
+ - `readCookie` — read a browser cookie value, `saveAs` a variable
270
+ - `readStorage` — read localStorage/sessionStorage, `saveAs` a variable
271
+ - `evaluate` — run JavaScript in the page, `saveAs` a variable
272
+ - `waitForNetwork` — wait for a network request matching filters
273
+ - `waitForCookie` — wait for a cookie to appear
274
+ - `goto` — navigate to a URL (e.g., trigger a login page)
275
+ - `solveCaptcha` — solve a CAPTCHA challenge
230
276
 
231
- - MUST use `goto({ url, captureNetwork })` to name navigation capture. `captureNetwork` is NOT supported on `open()`. In the CLI, this means `opensteer run page.goto --input-json ...`.
232
- - MUST query by capture first (`queryNetwork({ capture })`), then query all traffic to catch async requests.
233
- - MUST probe every discovered first-party API with transport tests. Do NOT just log URLs.
234
- - Only actions with `captureNetwork` are persisted at action time. Use `tagNetwork({ tag })` when you want to label already-persisted records for later lookup.
235
- - `queryNetwork({ ...filters })` always reads from the persisted store. It works the same whether the browser session is active or closed.
277
+ Each step can have a `capture` field to extract values from the response. The `outputs` field maps captured variables to `headers`, `query`, `params`, or `body` overrides applied to the request plan at execution time.
236
278
 
237
- Common mistakes:
279
+ #### Step 8d: Bind Auth Recipe to Plan
238
280
 
239
- - Do NOT pass headers as `{key: value}`. MUST use `[{name, value}]` arrays.
240
- - Do NOT pass body as a raw string. MUST wrap it in `{json: {...}}`, `{text: "..."}`, or `{base64: "..."}`.
241
- - Do NOT skip auth probing. If you find an OAuth endpoint, get a token and re-probe with it.
242
- - Do NOT treat "no data API found" as failure. It is a valid reverse-engineering conclusion that justifies DOM fallback.
243
- - Do NOT mix up recipes and auth recipes. They are separate registries and can reuse the same key/version independently.
281
+ Update the plan to reference the auth recipe:
244
282
 
245
- Additional guidance:
283
+ ```bash
284
+ opensteer run request-plan.get --workspace demo \
285
+ --input-json '{"key":"products.search","version":"v1"}'
286
+ ```
246
287
 
247
- - Capture the browser action first if authentication, cookies, or minted tokens may matter.
248
- - Probe with `direct-http` first. If it works, pass `transport: "direct-http"` to `inferRequestPlan` so the plan is portable. If it fails, fall back to `context-http`.
249
- - `inferRequestPlan()` throws if the key+version already exists. Catch the error or bump the version.
250
- - Inferred plans are immediately usable — `request.execute` works right after inference.
251
- - Use recipes when the request plan needs deterministic setup work. Use auth recipes when the setup is specifically auth refresh or login state.
252
- - Stay in the DOM workflow only when the rendered page itself is the deliverable. Move here when the request is the durable artifact.
288
+ Read the current payload, then write it back with the `auth.recipe` binding:
253
289
 
254
- ## When To Use Request Capture vs DOM Extraction
290
+ ```bash
291
+ opensteer run request-plan.write --workspace demo \
292
+ --input-json '{
293
+ "key":"products.search",
294
+ "version":"v1",
295
+ "payload":{
296
+ ...existing payload...,
297
+ "auth":{
298
+ "strategy":"bearer-token",
299
+ "recipe":{"key":"example.auth","version":"v1"},
300
+ "failurePolicy":{"on":"status","status":"401","action":"recover"}
301
+ }
302
+ }
303
+ }'
304
+ ```
305
+
306
+ The `failurePolicy` tells the plan to automatically re-run the auth recipe when a 401 is received.
307
+
308
+ #### Step 8e: Test Authenticated Plan
309
+
310
+ ```bash
311
+ opensteer run request.execute --workspace demo \
312
+ --input-json '{"key":"products.search","version":"v1","query":{"keyword":"laptop"}}'
313
+ ```
255
314
 
256
- See the task triage decision tree in SKILL.md for when to choose this workflow vs DOM extraction.
315
+ The auth recipe fires automatically before the request. If it still fails, inspect the token response shape and fix the recipe.
316
+
317
+ **EXIT CRITERIA:** `request.execute` returns valid data with the auth recipe attached.
318
+
319
+ ### Phase 9: Done
320
+
321
+ Close the browser session:
322
+
323
+ ```bash
324
+ opensteer close --workspace demo
325
+ ```
326
+
327
+ The plan persists in the workspace registry and (if cloud mode) in Convex. Future callers can replay it with `request.execute` without opening a browser.
257
328
 
258
329
  ## Error Recovery
259
330
 
260
- **`queryNetwork()` returns empty records:**
331
+ ### `request.execute` returns 400 Bad Request
332
+
333
+ 1. Read the plan: `request-plan.get`
334
+ 2. Compare the plan's `defaultQuery`, `defaultBody`, and `defaultHeaders` against the original captured request from Phase 2
335
+ 3. Identify the discrepancy — missing required parameter, wrong content-type, malformed body
336
+ 4. Fix the plan with `request-plan.write`
337
+ 5. Re-test with `request.execute`
338
+ 6. If still failing after 2 fix attempts, use `request.raw` to isolate which specific parameter causes the 400 — remove params one at a time
339
+
340
+ ### `request.execute` returns 401/403 Unauthorized
341
+
342
+ 1. Was auth classification validated in Phase 5? If not, go back to Phase 5.
343
+ 2. If auth is confirmed needed, enter Phase 8 (Auth Recipe).
344
+ 3. If an auth recipe exists but fails, inspect the token response — the token may have expired, the scope may be wrong, or the grant type may differ.
345
+
346
+ ### `request.execute` returns 404 Not Found
347
+
348
+ 1. The API path may have changed since capture. Re-capture traffic (Phase 1) and re-discover (Phase 2).
349
+ 2. Check if the URL uses path parameters that were hardcoded during inference. These may need to be templated.
350
+
351
+ ### `request.execute` returns 500 Server Error
352
+
353
+ This is the API server's problem, not a plan problem. Retry once. If persistent, document and report to the user.
354
+
355
+ ### `inferRequestPlan` throws "registry record already exists"
356
+
357
+ The key+version combination is already registered. Bump the version string (e.g., `v1` → `v2`).
358
+
359
+ ### `queryNetwork()` returns empty records
360
+
261
361
  - Verify `captureNetwork` was set on the action that triggered the request (not on `open()`).
262
- - Re-trigger the action. Records are auto-persisted, so re-triggering will capture new records.
362
+ - Re-trigger the action with `captureNetwork`. Records are auto-persisted on actions that opt in.
263
363
  - Broaden filters: try removing `tag` and querying by `hostname` or `path` instead.
264
364
  - Check that the request actually fired — some SPAs lazy-load data or use WebSocket instead of HTTP.
265
365
 
266
- **`rawRequest()` returns non-200 or errors:**
366
+ ### `rawRequest()` returns non-200
367
+
368
+ This is diagnostic information. Use it to decide transport and debug, not as a final answer.
267
369
  - If `direct-http` fails with 403/401: the API requires session state. Try `context-http`.
268
- - If `context-http` fails: the API may require specific cookies or tokens. Check for auth endpoints in the captured traffic.
370
+ - If `context-http` fails: the API may require specific cookies or tokens. Check for auth endpoints in captured traffic.
269
371
  - If the response body is empty: decode `response.body.data` with `Buffer.from(data, "base64").toString("utf8")` — the parsed `data` field may not be populated.
270
- - If the request times out: increase `timeoutMs` or check that the target server is reachable.
271
372
 
272
- **`waitForNetwork()` times out:**
273
- - `waitForNetwork()` only matches records that appear AFTER the call starts. If the request already fired, it will not be found.
274
- - Ensure the triggering action (click, scroll, etc.) happens AFTER calling `waitForNetwork()`, or use `queryNetwork()` instead for already-captured records.
275
- - Increase `timeoutMs` if the request is slow.
373
+ ### `waitForNetwork()` times out
276
374
 
277
- **`inferRequestPlan()` throws "registry record already exists":**
278
- - The key+version combination is already registered. Either use a new version string or catch the error.
375
+ - `waitForNetwork()` only matches records that appear AFTER the call starts. If the request already fired, use `queryNetwork()` instead.
376
+ - Ensure the triggering action happens AFTER calling `waitForNetwork()`.
279
377
 
280
- **`tagNetwork()` returns `taggedCount: 0`:**
281
- - No records matched the filters. Verify the records exist with `queryNetwork()` using the same filters first.
378
+ ## Input Formats
379
+
380
+ `rawRequest` and recipe steps expect specific shapes:
381
+
382
+ - `headers`: array of `[{ name, value }]`, not `{ key: value }`. Exception: recipe step `request` fields accept `{ key: value }` objects.
383
+ - `body`: one of `{ json: { ... } }`, `{ text: "..." }`, or `{ base64: "..." }`. Not raw strings.
384
+ - `request.execute` input includes `key` inside the JSON object. The SDK convenience wrapper `opensteer.request(key, input)` adds that for you.
385
+
386
+ ## Practical Guidance
387
+
388
+ Mandatory steps:
389
+ - MUST use `goto({ url, captureNetwork })` to name navigation capture. `captureNetwork` is NOT supported on `open()`.
390
+ - MUST query by capture first, then query all traffic to catch async requests.
391
+ - MUST probe every discovered first-party API with transport tests. Do NOT just log URLs.
392
+ - The deliverable is a persisted plan. `rawRequest()` output is never the final answer.
393
+
394
+ Common mistakes:
395
+ - Stopping at `rawRequest` output and returning it to the user. Always proceed to `inferRequestPlan` and `request.execute`.
396
+ - Trusting inferred auth metadata without validation. Always run Phase 5.
397
+ - Passing headers as `{key: value}` to `rawRequest`. MUST use `[{name, value}]` arrays.
398
+ - Passing body as a raw string to `rawRequest`. MUST wrap in `{json: {...}}`, `{text: "..."}`, or `{base64: "..."}`.
399
+ - Skipping parameter annotation. Variable params should be documented in the plan's `parameters` field.
400
+ - Not closing the browser after completing the pipeline. Always run `opensteer close` when done.