opensteer 0.8.9 → 0.8.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{chunk-RO6WAWWG.js → chunk-F3X6UOEN.js} +1959 -215
- package/dist/chunk-F3X6UOEN.js.map +1 -0
- package/dist/cli/bin.cjs +2165 -149
- package/dist/cli/bin.cjs.map +1 -1
- package/dist/cli/bin.js +281 -8
- package/dist/cli/bin.js.map +1 -1
- package/dist/index.cjs +170 -26
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +21 -6
- package/dist/index.d.ts +21 -6
- package/dist/index.js +2 -2
- package/package.json +1 -1
- package/skills/opensteer/SKILL.md +33 -14
- package/skills/opensteer/references/request-workflow.md +312 -193
- package/skills/opensteer/references/sdk-reference.md +102 -14
- package/skills/recorder/SKILL.md +54 -0
- package/skills/recorder/references/recorder-reference.md +71 -0
- package/dist/chunk-RO6WAWWG.js.map +0 -1
|
@@ -1,22 +1,10 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Request Plan Pipeline
|
|
2
2
|
|
|
3
3
|
If you haven't decided whether this workflow applies, see the task triage in SKILL.md.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## The Deliverable
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
- [Critical Rules](#critical-rules)
|
|
10
|
-
- [Standard Loop](#standard-loop)
|
|
11
|
-
- [Transport Selection](#transport-selection)
|
|
12
|
-
- [SDK Flow](#sdk-flow)
|
|
13
|
-
- [CLI Equivalents](#cli-equivalents)
|
|
14
|
-
- [Transport Probing](#transport-probing)
|
|
15
|
-
- [Auth Token Acquisition](#auth-token-acquisition)
|
|
16
|
-
- [Input Formats](#input-formats)
|
|
17
|
-
- [Practical Guidance](#practical-guidance)
|
|
18
|
-
- [When To Use Request Capture vs DOM Extraction](#when-to-use-request-capture-vs-dom-extraction)
|
|
19
|
-
- [Error Recovery](#error-recovery)
|
|
7
|
+
The deliverable is a **persisted request plan** that works via `request.execute`. `rawRequest()` is a diagnostic probe — its output is never the final answer. You are not done until `request.execute` returns valid data from a stored plan.
|
|
20
8
|
|
|
21
9
|
## Critical Rules
|
|
22
10
|
|
|
@@ -26,19 +14,6 @@ Use this workflow when the deliverable is a custom API, a replayable request pla
|
|
|
26
14
|
4. `clearNetwork()` permanently removes records with tombstoning. Cleared records cannot be resurrected by late-arriving browser events.
|
|
27
15
|
5. `waitForNetwork()` watches for NEW records only. It snapshots existing records and polls for ones that were not present at the start. It does NOT return historical matches.
|
|
28
16
|
|
|
29
|
-
## Standard Loop
|
|
30
|
-
|
|
31
|
-
1. Trigger the real browser action that causes the request inside a stable workspace.
|
|
32
|
-
2. Name the important navigation or interaction with `captureNetwork` on the action itself.
|
|
33
|
-
3. Inspect the captured traffic with `queryNetwork()` and isolate the relevant records.
|
|
34
|
-
4. Use `tagNetwork()` to label interesting records for later lookup if you want extra manual labels.
|
|
35
|
-
5. Probe the request with `rawRequest()` — try `direct-http` first, then `context-http`.
|
|
36
|
-
6. Infer a request plan from the probed record — pass `transport` if you proved portability.
|
|
37
|
-
7. Add recipes or auth recipes if replay needs deterministic setup.
|
|
38
|
-
8. Replay the plan from code — works immediately, no extra steps.
|
|
39
|
-
|
|
40
|
-
This workflow should carry equal weight with DOM automation. Use it whenever the browser page is only the launcher for the real target request.
|
|
41
|
-
|
|
42
17
|
## Transport Selection
|
|
43
18
|
|
|
44
19
|
- `direct-http`: the request is replayable without a browser.
|
|
@@ -48,234 +23,378 @@ This workflow should carry equal weight with DOM automation. Use it whenever the
|
|
|
48
23
|
|
|
49
24
|
When in doubt, start with browser-backed capture first. Opensteer treats browser-backed replay as a first-class path, not a fallback.
|
|
50
25
|
|
|
51
|
-
##
|
|
26
|
+
## Pipeline Phases
|
|
52
27
|
|
|
53
|
-
|
|
28
|
+
Work through each phase in order. Do NOT skip phases. Each phase has exit criteria — verify them before proceeding.
|
|
54
29
|
|
|
55
|
-
|
|
56
|
-
await opensteer.open();
|
|
57
|
-
await opensteer.goto({
|
|
58
|
-
url: "https://example.com/app",
|
|
59
|
-
captureNetwork: "page-load",
|
|
60
|
-
});
|
|
30
|
+
### Phase 1: Capture
|
|
61
31
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
32
|
+
Trigger the real browser action that causes the request. Name the capture.
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
opensteer open https://example.com/app --workspace demo
|
|
36
|
+
opensteer run page.goto --workspace demo \
|
|
37
|
+
--input-json '{"url":"https://example.com/app","captureNetwork":"page-load"}'
|
|
66
38
|
```
|
|
67
39
|
|
|
68
|
-
|
|
40
|
+
For interactions that trigger API calls (search, filter, load-more):
|
|
69
41
|
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
includeBodies: true,
|
|
74
|
-
limit: 20,
|
|
75
|
-
});
|
|
42
|
+
```bash
|
|
43
|
+
opensteer run dom.click --workspace demo \
|
|
44
|
+
--input-json '{"target":{"kind":"description","description":"load products"},"captureNetwork":"products-load"}'
|
|
76
45
|
```
|
|
77
46
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
},
|
|
88
|
-
});
|
|
89
|
-
// If direct-http returns 200, the API is portable (no browser needed).
|
|
90
|
-
// If it fails, try context-http — the API needs browser session state.
|
|
47
|
+
**EXIT CRITERIA:** You have at least one named capture. If `queryNetwork` returns empty after capture, see Error Recovery.
|
|
48
|
+
|
|
49
|
+
### Phase 2: Discover
|
|
50
|
+
|
|
51
|
+
Query captured traffic to isolate the API calls worth replaying. Ignore static assets, analytics, and third-party scripts.
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
opensteer run network.query --workspace demo \
|
|
55
|
+
--input-json '{"capture":"products-load","includeBodies":true,"limit":20}'
|
|
91
56
|
```
|
|
92
57
|
|
|
93
|
-
|
|
58
|
+
Examine the results. Look for first-party JSON APIs — requests returning `application/json` with data relevant to the task.
|
|
94
59
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
transport: "direct-http", // use the transport you proved works
|
|
101
|
-
});
|
|
60
|
+
If the first query is too broad, filter by hostname, path, or method:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
opensteer run network.query --workspace demo \
|
|
64
|
+
--input-json '{"capture":"products-load","hostname":"api.example.com","method":"GET","includeBodies":true,"limit":10}'
|
|
102
65
|
```
|
|
103
66
|
|
|
104
|
-
|
|
67
|
+
**EXIT CRITERIA:** You have identified at least one candidate API URL with its method, recordId, and response shape.
|
|
68
|
+
|
|
69
|
+
### Phase 3: Probe (Diagnostic Only)
|
|
105
70
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
71
|
+
`rawRequest()` is a diagnostic tool. Use it to determine:
|
|
72
|
+
1. Which transport works (`direct-http` vs `context-http`)
|
|
73
|
+
2. Whether the API returns the expected data shape
|
|
74
|
+
3. Whether auth headers are actually required
|
|
75
|
+
|
|
76
|
+
`rawRequest()` output is NOT the deliverable. Do NOT return rawRequest results to the user as the final answer. Always proceed to Phase 4.
|
|
77
|
+
|
|
78
|
+
Test portability — try `direct-http` first:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
opensteer run request.raw --workspace demo \
|
|
82
|
+
--input-json '{"transport":"direct-http","url":"https://api.example.com/products","method":"GET"}'
|
|
110
83
|
```
|
|
111
84
|
|
|
112
|
-
|
|
85
|
+
If `direct-http` returns 200, the API is portable. If it fails (403/401), try `context-http`:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
opensteer run request.raw --workspace demo \
|
|
89
|
+
--input-json '{"transport":"context-http","url":"https://api.example.com/products","method":"GET"}'
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
**EXIT CRITERIA:** You know which transport works and have a successful response. Note the `recordId` from the probe response — you will use it in Phase 4.
|
|
93
|
+
|
|
94
|
+
### Phase 4: Infer Plan
|
|
113
95
|
|
|
114
|
-
|
|
96
|
+
Create a request plan from the probed record. Pass the `transport` you proved works.
|
|
115
97
|
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
});
|
|
98
|
+
```bash
|
|
99
|
+
opensteer run request-plan.infer --workspace demo \
|
|
100
|
+
--input-json '{"recordId":"<recordId-from-phase-3>","key":"products.search","version":"v1","transport":"direct-http"}'
|
|
120
101
|
```
|
|
121
102
|
|
|
122
|
-
|
|
103
|
+
If you proved `direct-http` works, always pass `transport: "direct-http"` so the plan is portable.
|
|
104
|
+
|
|
105
|
+
If `inferRequestPlan` throws "registry record already exists", bump the version (e.g., `v2`).
|
|
106
|
+
|
|
107
|
+
**EXIT CRITERIA:** Plan is persisted. You can verify with `request-plan.get`.
|
|
123
108
|
|
|
124
|
-
|
|
125
|
-
await opensteer.runRecipe({
|
|
126
|
-
key: "products.setup",
|
|
127
|
-
});
|
|
109
|
+
### Phase 5: Validate Auth Classification
|
|
128
110
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
111
|
+
`inferRequestPlan` records auth metadata by observing headers on the captured request. This is **often wrong**. If the browser sent an `Authorization` header, the plan records `auth.strategy: "bearer-token"` even if the API works without auth.
|
|
112
|
+
|
|
113
|
+
**MANDATORY VALIDATION:**
|
|
114
|
+
|
|
115
|
+
1. Read the inferred plan:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
opensteer run request-plan.get --workspace demo \
|
|
119
|
+
--input-json '{"key":"products.search","version":"v1"}'
|
|
132
120
|
```
|
|
133
121
|
|
|
134
|
-
|
|
122
|
+
2. Check the `auth` field in `payload`. If `auth` is absent or `undefined`, auth is not detected — skip to Phase 6.
|
|
123
|
+
|
|
124
|
+
3. If `auth.strategy` is set, test whether the API actually needs it. Run a raw request to the same URL with NO auth headers:
|
|
135
125
|
|
|
136
126
|
```bash
|
|
137
|
-
opensteer open https://example.com/app --workspace demo
|
|
138
|
-
opensteer run page.goto --workspace demo \
|
|
139
|
-
--input-json '{"url":"https://example.com/app","captureNetwork":"page-load"}'
|
|
140
|
-
opensteer click --workspace demo --description "load products"
|
|
141
|
-
# or with captureNetwork: opensteer run dom.click --workspace demo \
|
|
142
|
-
# --input-json '{"target":{"kind":"description","description":"load products"},"captureNetwork":"products-load"}'
|
|
143
|
-
opensteer run network.query --workspace demo \
|
|
144
|
-
--input-json '{"capture":"products-load","includeBodies":true,"limit":20}'
|
|
145
127
|
opensteer run request.raw --workspace demo \
|
|
146
|
-
--input-json '{"transport":"direct-http","url":"
|
|
147
|
-
opensteer run request-plan.infer --workspace demo \
|
|
148
|
-
--input-json '{"recordId":"rec_123","key":"products.search","version":"v1","transport":"direct-http"}'
|
|
149
|
-
opensteer run request.execute --workspace demo \
|
|
150
|
-
--input-json '{"key":"products.search","query":{"q":"laptop"}}'
|
|
128
|
+
--input-json '{"transport":"direct-http","url":"<the-api-url>","method":"GET"}'
|
|
151
129
|
```
|
|
152
130
|
|
|
153
|
-
|
|
131
|
+
4. If it returns 200 without auth headers, auth is **spurious** — the browser attached a token the API doesn't enforce. Rewrite the plan with corrected auth:
|
|
154
132
|
|
|
155
|
-
|
|
133
|
+
```bash
|
|
134
|
+
opensteer run request-plan.write --workspace demo \
|
|
135
|
+
--input-json '{
|
|
136
|
+
"key":"products.search",
|
|
137
|
+
"version":"v1",
|
|
138
|
+
"tags":["products","search"],
|
|
139
|
+
"provenance":{"source":"manual","notes":"Auth removed — API is public, bearer token was incidental."},
|
|
140
|
+
"payload":{
|
|
141
|
+
...existing payload with auth field removed...
|
|
142
|
+
}
|
|
143
|
+
}'
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Copy the full existing `payload` from `request-plan.get`, remove or null out the `auth` field, and write it back.
|
|
147
|
+
|
|
148
|
+
5. If the no-auth probe returns 401/403, auth IS required. Keep the auth classification and proceed. You will create an auth recipe in Phase 8 after testing the plan.
|
|
149
|
+
|
|
150
|
+
**EXIT CRITERIA:** The plan's `auth` field accurately reflects whether auth is required.
|
|
151
|
+
|
|
152
|
+
### Phase 6: Annotate Parameters
|
|
153
|
+
|
|
154
|
+
`inferRequestPlan` dumps all query and body parameters into `defaultQuery`/`defaultBody` without distinguishing variable from fixed.
|
|
155
|
+
|
|
156
|
+
1. Read the plan with `request-plan.get`.
|
|
156
157
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
url: discoveredUrl,
|
|
161
|
-
method: "GET",
|
|
162
|
-
});
|
|
158
|
+
2. Examine each parameter in `defaultQuery`:
|
|
159
|
+
- **Variable:** values that change per invocation — search terms, page numbers, offsets, dates, user-specific IDs
|
|
160
|
+
- **Fixed:** values constant for this API — site keys, platform identifiers, API versions, channel strings
|
|
163
161
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
162
|
+
3. Rewrite the plan with the `parameters` field annotating variable inputs:
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
opensteer run request-plan.write --workspace demo \
|
|
166
|
+
--input-json '{
|
|
167
|
+
"key":"products.search",
|
|
168
|
+
"version":"v1",
|
|
169
|
+
"payload":{
|
|
170
|
+
...existing payload...,
|
|
171
|
+
"parameters":[
|
|
172
|
+
{"name":"keyword","in":"query","required":true,"description":"Search term"},
|
|
173
|
+
{"name":"count","in":"query","defaultValue":"24","description":"Results per page"},
|
|
174
|
+
{"name":"offset","in":"query","defaultValue":"0","description":"Pagination offset"}
|
|
175
|
+
]
|
|
176
|
+
}
|
|
177
|
+
}'
|
|
169
178
|
```
|
|
170
179
|
|
|
171
|
-
|
|
180
|
+
Variable params remain in `defaultQuery` as initial values. The `parameters` field documents which ones a caller should override via `request.execute` input.
|
|
181
|
+
|
|
182
|
+
**EXIT CRITERIA:** The plan's `parameters` field lists all variable inputs with descriptions.
|
|
183
|
+
|
|
184
|
+
### Phase 7: Test Plan
|
|
172
185
|
|
|
173
|
-
|
|
186
|
+
Execute the plan through `request.execute`, NOT `rawRequest`:
|
|
174
187
|
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
key: "products.search.portable",
|
|
179
|
-
version: "v1",
|
|
180
|
-
transport: "direct-http",
|
|
181
|
-
});
|
|
188
|
+
```bash
|
|
189
|
+
opensteer run request.execute --workspace demo \
|
|
190
|
+
--input-json '{"key":"products.search","version":"v1","query":{"keyword":"laptop","count":"10"}}'
|
|
182
191
|
```
|
|
183
192
|
|
|
193
|
+
**GATE:**
|
|
194
|
+
- If `request.execute` returns valid data → proceed to Phase 9 (Done).
|
|
195
|
+
- If `request.execute` returns 401/403 and Phase 5 confirmed auth is required → proceed to Phase 8 (Auth Recipe).
|
|
196
|
+
- If `request.execute` fails with another error → see Error Recovery.
|
|
197
|
+
|
|
198
|
+
### Phase 8: Auth Recipe (Conditional)
|
|
199
|
+
|
|
200
|
+
Enter this phase ONLY if Phase 5 confirmed auth is genuinely required AND Phase 7 failed with 401/403.
|
|
201
|
+
|
|
202
|
+
#### Step 8a: Discover Auth Endpoint
|
|
203
|
+
|
|
204
|
+
Search captured traffic for OAuth, token, or login endpoints:
|
|
205
|
+
|
|
184
206
|
```bash
|
|
185
|
-
opensteer run
|
|
186
|
-
--input-json '{"
|
|
207
|
+
opensteer run network.query --workspace demo \
|
|
208
|
+
--input-json '{"path":"/oauth","includeBodies":true,"limit":10}'
|
|
209
|
+
opensteer run network.query --workspace demo \
|
|
210
|
+
--input-json '{"path":"/token","includeBodies":true,"limit":10}'
|
|
211
|
+
opensteer run network.query --workspace demo \
|
|
212
|
+
--input-json '{"path":"/auth","includeBodies":true,"limit":10}'
|
|
187
213
|
```
|
|
188
214
|
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
if (!body) {
|
|
204
|
-
throw new Error("Token response had no body");
|
|
205
|
-
}
|
|
206
|
-
parsed = JSON.parse(Buffer.from(body.data, "base64").toString("utf8"));
|
|
207
|
-
}
|
|
208
|
-
|
|
209
|
-
const token = String((parsed as { access_token: unknown }).access_token);
|
|
210
|
-
|
|
211
|
-
const authed = await opensteer.rawRequest({
|
|
212
|
-
transport: "direct-http",
|
|
213
|
-
url: "https://example.com/api/products",
|
|
214
|
-
method: "GET",
|
|
215
|
-
headers: [{ name: "Authorization", value: `Bearer ${token}` }],
|
|
216
|
-
});
|
|
215
|
+
Examine responses to find the endpoint that returns an access token.
|
|
216
|
+
|
|
217
|
+
#### Step 8b: Probe Auth Endpoint
|
|
218
|
+
|
|
219
|
+
Test the auth endpoint with `request.raw`:
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
opensteer run request.raw --workspace demo \
|
|
223
|
+
--input-json '{
|
|
224
|
+
"transport":"direct-http",
|
|
225
|
+
"url":"https://example.com/api/oauth/token",
|
|
226
|
+
"method":"POST",
|
|
227
|
+
"body":{"json":{"grant_type":"client_credentials"}}
|
|
228
|
+
}'
|
|
217
229
|
```
|
|
218
230
|
|
|
219
|
-
|
|
231
|
+
Verify it returns a token. Note the response shape (e.g., `{ "access_token": "..." }`).
|
|
220
232
|
|
|
221
|
-
|
|
233
|
+
#### Step 8c: Create Auth Recipe
|
|
222
234
|
|
|
223
|
-
|
|
224
|
-
- `body`: one of `{ json: { ... } }`, `{ text: "..." }`, or `{ base64: "..." }`. Not raw strings.
|
|
225
|
-
- `request.execute` semantic input includes `key` inside the JSON object. The SDK convenience wrapper `opensteer.request(key, input)` adds that for you.
|
|
235
|
+
Write an auth recipe that acquires a fresh token and maps it to request headers:
|
|
226
236
|
|
|
227
|
-
|
|
237
|
+
```bash
|
|
238
|
+
opensteer run auth-recipe.write --workspace demo \
|
|
239
|
+
--input-json '{
|
|
240
|
+
"key":"example.auth",
|
|
241
|
+
"version":"v1",
|
|
242
|
+
"payload":{
|
|
243
|
+
"description":"Acquire bearer token for example.com API",
|
|
244
|
+
"steps":[
|
|
245
|
+
{
|
|
246
|
+
"kind":"directRequest",
|
|
247
|
+
"request":{
|
|
248
|
+
"url":"https://example.com/api/oauth/token",
|
|
249
|
+
"transport":"direct-http",
|
|
250
|
+
"method":"POST",
|
|
251
|
+
"body":{"json":{"grant_type":"client_credentials"}}
|
|
252
|
+
},
|
|
253
|
+
"capture":{
|
|
254
|
+
"bodyJsonPointer":{"pointer":"/access_token","saveAs":"token"}
|
|
255
|
+
}
|
|
256
|
+
}
|
|
257
|
+
],
|
|
258
|
+
"outputs":{
|
|
259
|
+
"headers":{"Authorization":"Bearer {{token}}"}
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
}'
|
|
263
|
+
```
|
|
228
264
|
|
|
229
|
-
|
|
265
|
+
**Recipe step types you can use:**
|
|
266
|
+
- `directRequest` — HTTP request outside the browser (portable, no session needed)
|
|
267
|
+
- `sessionRequest` — HTTP request using browser session state (cookies, etc.)
|
|
268
|
+
- `request` — generic request step
|
|
269
|
+
- `readCookie` — read a browser cookie value, `saveAs` a variable
|
|
270
|
+
- `readStorage` — read localStorage/sessionStorage, `saveAs` a variable
|
|
271
|
+
- `evaluate` — run JavaScript in the page, `saveAs` a variable
|
|
272
|
+
- `waitForNetwork` — wait for a network request matching filters
|
|
273
|
+
- `waitForCookie` — wait for a cookie to appear
|
|
274
|
+
- `goto` — navigate to a URL (e.g., trigger a login page)
|
|
275
|
+
- `solveCaptcha` — solve a CAPTCHA challenge
|
|
230
276
|
|
|
231
|
-
|
|
232
|
-
- MUST query by capture first (`queryNetwork({ capture })`), then query all traffic to catch async requests.
|
|
233
|
-
- MUST probe every discovered first-party API with transport tests. Do NOT just log URLs.
|
|
234
|
-
- Only actions with `captureNetwork` are persisted at action time. Use `tagNetwork({ tag })` when you want to label already-persisted records for later lookup.
|
|
235
|
-
- `queryNetwork({ ...filters })` always reads from the persisted store. It works the same whether the browser session is active or closed.
|
|
277
|
+
Each step can have a `capture` field to extract values from the response. The `outputs` field maps captured variables to `headers`, `query`, `params`, or `body` overrides applied to the request plan at execution time.
|
|
236
278
|
|
|
237
|
-
|
|
279
|
+
#### Step 8d: Bind Auth Recipe to Plan
|
|
238
280
|
|
|
239
|
-
|
|
240
|
-
- Do NOT pass body as a raw string. MUST wrap it in `{json: {...}}`, `{text: "..."}`, or `{base64: "..."}`.
|
|
241
|
-
- Do NOT skip auth probing. If you find an OAuth endpoint, get a token and re-probe with it.
|
|
242
|
-
- Do NOT treat "no data API found" as failure. It is a valid reverse-engineering conclusion that justifies DOM fallback.
|
|
243
|
-
- Do NOT mix up recipes and auth recipes. They are separate registries and can reuse the same key/version independently.
|
|
281
|
+
Update the plan to reference the auth recipe:
|
|
244
282
|
|
|
245
|
-
|
|
283
|
+
```bash
|
|
284
|
+
opensteer run request-plan.get --workspace demo \
|
|
285
|
+
--input-json '{"key":"products.search","version":"v1"}'
|
|
286
|
+
```
|
|
246
287
|
|
|
247
|
-
|
|
248
|
-
- Probe with `direct-http` first. If it works, pass `transport: "direct-http"` to `inferRequestPlan` so the plan is portable. If it fails, fall back to `context-http`.
|
|
249
|
-
- `inferRequestPlan()` throws if the key+version already exists. Catch the error or bump the version.
|
|
250
|
-
- Inferred plans are immediately usable — `request.execute` works right after inference.
|
|
251
|
-
- Use recipes when the request plan needs deterministic setup work. Use auth recipes when the setup is specifically auth refresh or login state.
|
|
252
|
-
- Stay in the DOM workflow only when the rendered page itself is the deliverable. Move here when the request is the durable artifact.
|
|
288
|
+
Read the current payload, then write it back with the `auth.recipe` binding:
|
|
253
289
|
|
|
254
|
-
|
|
290
|
+
```bash
|
|
291
|
+
opensteer run request-plan.write --workspace demo \
|
|
292
|
+
--input-json '{
|
|
293
|
+
"key":"products.search",
|
|
294
|
+
"version":"v1",
|
|
295
|
+
"payload":{
|
|
296
|
+
...existing payload...,
|
|
297
|
+
"auth":{
|
|
298
|
+
"strategy":"bearer-token",
|
|
299
|
+
"recipe":{"key":"example.auth","version":"v1"},
|
|
300
|
+
"failurePolicy":{"on":"status","status":"401","action":"recover"}
|
|
301
|
+
}
|
|
302
|
+
}
|
|
303
|
+
}'
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
The `failurePolicy` tells the plan to automatically re-run the auth recipe when a 401 is received.
|
|
307
|
+
|
|
308
|
+
#### Step 8e: Test Authenticated Plan
|
|
309
|
+
|
|
310
|
+
```bash
|
|
311
|
+
opensteer run request.execute --workspace demo \
|
|
312
|
+
--input-json '{"key":"products.search","version":"v1","query":{"keyword":"laptop"}}'
|
|
313
|
+
```
|
|
255
314
|
|
|
256
|
-
|
|
315
|
+
The auth recipe fires automatically before the request. If it still fails, inspect the token response shape and fix the recipe.
|
|
316
|
+
|
|
317
|
+
**EXIT CRITERIA:** `request.execute` returns valid data with the auth recipe attached.
|
|
318
|
+
|
|
319
|
+
### Phase 9: Done
|
|
320
|
+
|
|
321
|
+
Close the browser session:
|
|
322
|
+
|
|
323
|
+
```bash
|
|
324
|
+
opensteer close --workspace demo
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
The plan persists in the workspace registry and (if cloud mode) in Convex. Future callers can replay it with `request.execute` without opening a browser.
|
|
257
328
|
|
|
258
329
|
## Error Recovery
|
|
259
330
|
|
|
260
|
-
|
|
331
|
+
### `request.execute` returns 400 Bad Request
|
|
332
|
+
|
|
333
|
+
1. Read the plan: `request-plan.get`
|
|
334
|
+
2. Compare the plan's `defaultQuery`, `defaultBody`, and `defaultHeaders` against the original captured request from Phase 2
|
|
335
|
+
3. Identify the discrepancy — missing required parameter, wrong content-type, malformed body
|
|
336
|
+
4. Fix the plan with `request-plan.write`
|
|
337
|
+
5. Re-test with `request.execute`
|
|
338
|
+
6. If still failing after 2 fix attempts, use `request.raw` to isolate which specific parameter causes the 400 — remove params one at a time
|
|
339
|
+
|
|
340
|
+
### `request.execute` returns 401/403 Unauthorized
|
|
341
|
+
|
|
342
|
+
1. Was auth classification validated in Phase 5? If not, go back to Phase 5.
|
|
343
|
+
2. If auth is confirmed needed, enter Phase 8 (Auth Recipe).
|
|
344
|
+
3. If an auth recipe exists but fails, inspect the token response — the token may have expired, the scope may be wrong, or the grant type may differ.
|
|
345
|
+
|
|
346
|
+
### `request.execute` returns 404 Not Found
|
|
347
|
+
|
|
348
|
+
1. The API path may have changed since capture. Re-capture traffic (Phase 1) and re-discover (Phase 2).
|
|
349
|
+
2. Check if the URL uses path parameters that were hardcoded during inference. These may need to be templated.
|
|
350
|
+
|
|
351
|
+
### `request.execute` returns 500 Server Error
|
|
352
|
+
|
|
353
|
+
This is the API server's problem, not a plan problem. Retry once. If persistent, document and report to the user.
|
|
354
|
+
|
|
355
|
+
### `inferRequestPlan` throws "registry record already exists"
|
|
356
|
+
|
|
357
|
+
The key+version combination is already registered. Bump the version string (e.g., `v1` → `v2`).
|
|
358
|
+
|
|
359
|
+
### `queryNetwork()` returns empty records
|
|
360
|
+
|
|
261
361
|
- Verify `captureNetwork` was set on the action that triggered the request (not on `open()`).
|
|
262
|
-
- Re-trigger the action
|
|
362
|
+
- Re-trigger the action with `captureNetwork`. Records are auto-persisted on actions that opt in.
|
|
263
363
|
- Broaden filters: try removing `tag` and querying by `hostname` or `path` instead.
|
|
264
364
|
- Check that the request actually fired — some SPAs lazy-load data or use WebSocket instead of HTTP.
|
|
265
365
|
|
|
266
|
-
|
|
366
|
+
### `rawRequest()` returns non-200
|
|
367
|
+
|
|
368
|
+
This is diagnostic information. Use it to decide transport and debug, not as a final answer.
|
|
267
369
|
- If `direct-http` fails with 403/401: the API requires session state. Try `context-http`.
|
|
268
|
-
- If `context-http` fails: the API may require specific cookies or tokens. Check for auth endpoints in
|
|
370
|
+
- If `context-http` fails: the API may require specific cookies or tokens. Check for auth endpoints in captured traffic.
|
|
269
371
|
- If the response body is empty: decode `response.body.data` with `Buffer.from(data, "base64").toString("utf8")` — the parsed `data` field may not be populated.
|
|
270
|
-
- If the request times out: increase `timeoutMs` or check that the target server is reachable.
|
|
271
372
|
|
|
272
|
-
|
|
273
|
-
- `waitForNetwork()` only matches records that appear AFTER the call starts. If the request already fired, it will not be found.
|
|
274
|
-
- Ensure the triggering action (click, scroll, etc.) happens AFTER calling `waitForNetwork()`, or use `queryNetwork()` instead for already-captured records.
|
|
275
|
-
- Increase `timeoutMs` if the request is slow.
|
|
373
|
+
### `waitForNetwork()` times out
|
|
276
374
|
|
|
277
|
-
|
|
278
|
-
-
|
|
375
|
+
- `waitForNetwork()` only matches records that appear AFTER the call starts. If the request already fired, use `queryNetwork()` instead.
|
|
376
|
+
- Ensure the triggering action happens AFTER calling `waitForNetwork()`.
|
|
279
377
|
|
|
280
|
-
|
|
281
|
-
|
|
378
|
+
## Input Formats
|
|
379
|
+
|
|
380
|
+
`rawRequest` and recipe steps expect specific shapes:
|
|
381
|
+
|
|
382
|
+
- `headers`: array of `[{ name, value }]`, not `{ key: value }`. Exception: recipe step `request` fields accept `{ key: value }` objects.
|
|
383
|
+
- `body`: one of `{ json: { ... } }`, `{ text: "..." }`, or `{ base64: "..." }`. Not raw strings.
|
|
384
|
+
- `request.execute` input includes `key` inside the JSON object. The SDK convenience wrapper `opensteer.request(key, input)` adds that for you.
|
|
385
|
+
|
|
386
|
+
## Practical Guidance
|
|
387
|
+
|
|
388
|
+
Mandatory steps:
|
|
389
|
+
- MUST use `goto({ url, captureNetwork })` to name navigation capture. `captureNetwork` is NOT supported on `open()`.
|
|
390
|
+
- MUST query by capture first, then query all traffic to catch async requests.
|
|
391
|
+
- MUST probe every discovered first-party API with transport tests. Do NOT just log URLs.
|
|
392
|
+
- The deliverable is a persisted plan. `rawRequest()` output is never the final answer.
|
|
393
|
+
|
|
394
|
+
Common mistakes:
|
|
395
|
+
- Stopping at `rawRequest` output and returning it to the user. Always proceed to `inferRequestPlan` and `request.execute`.
|
|
396
|
+
- Trusting inferred auth metadata without validation. Always run Phase 5.
|
|
397
|
+
- Passing headers as `{key: value}` to `rawRequest`. MUST use `[{name, value}]` arrays.
|
|
398
|
+
- Passing body as a raw string to `rawRequest`. MUST wrap in `{json: {...}}`, `{text: "..."}`, or `{base64: "..."}`.
|
|
399
|
+
- Skipping parameter annotation. Variable params should be documented in the plan's `parameters` field.
|
|
400
|
+
- Not closing the browser after completing the pipeline. Always run `opensteer close` when done.
|