@scrapa/sdk 0.2.2 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -17
- package/dist/client.js +2 -2
- package/dist/client.js.map +1 -1
- package/dist/errors.js +1 -1
- package/dist/errors.js.map +1 -1
- package/dist/generated/contract.d.ts +66 -10
- package/dist/generated/contract.d.ts.map +1 -1
- package/dist/generated/contract.js +24 -9
- package/dist/generated/contract.js.map +1 -1
- package/dist/generated/operations.d.ts +77 -45
- package/dist/generated/operations.d.ts.map +1 -1
- package/dist/generated/operations.js +89 -43
- package/dist/generated/operations.js.map +1 -1
- package/dist/generated/params.d.ts +250 -98
- package/dist/generated/params.d.ts.map +1 -1
- package/dist/generated/retry.d.ts +2 -2
- package/dist/generated/retry.js +3 -3
- package/dist/generated/transport.js +2 -2
- package/package.json +2 -2
- package/src/client.ts +2 -2
- package/src/errors.ts +1 -1
- package/src/generated/contract.ts +81 -9
- package/src/generated/operations.ts +109 -43
- package/src/generated/params.ts +252 -98
- package/src/generated/retry.ts +3 -3
- package/src/generated/transport.ts +2 -2
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @scrapa/sdk
|
|
2
2
|
|
|
3
|
-
The official TypeScript client for [scrapa.app](https://scrapa.app)
|
|
3
|
+
The official TypeScript client for [scrapa.app](https://scrapa.app): read any web page as
|
|
4
4
|
markdown, HTML, text, links, a screenshot or a PDF, through a plain HTTP request or a real
|
|
5
5
|
browser, whichever the page turns out to need. Search the web through the same key, and hold
|
|
6
6
|
a real Chromium your own Playwright or Puppeteer drives.
|
|
@@ -20,13 +20,13 @@ console.log(page.markdown);
|
|
|
20
20
|
|
|
21
21
|
Every method, every argument and every field is generated from the API's own definitions, so
|
|
22
22
|
what your editor autocompletes is what the server accepts. Node 20+, Bun, Deno, workers and
|
|
23
|
-
the browser
|
|
23
|
+
the browser. It runs anywhere `fetch` exists.
|
|
24
24
|
|
|
25
25
|
## Reading a page
|
|
26
26
|
|
|
27
27
|
`scrape()` takes a URL and gives you the page. `engine` defaults to `auto`: a plain HTTP
|
|
28
28
|
request first, re-read in a real Chromium only when the body turns out to be a JavaScript
|
|
29
|
-
shell
|
|
29
|
+
shell. That is the difference between 1 credit and 5.
|
|
30
30
|
|
|
31
31
|
```ts
|
|
32
32
|
const page = await scrapa.scrape({
|
|
@@ -34,11 +34,22 @@ const page = await scrapa.scrape({
|
|
|
34
34
|
formats: ["markdown", "links"],
|
|
35
35
|
});
|
|
36
36
|
|
|
37
|
-
page.engine; // "fetch"
|
|
37
|
+
page.engine; // "fetch": this one did not need a browser
|
|
38
38
|
page.links; // every absolute link, in document order
|
|
39
39
|
page.metadata.title;
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
+
`markdown` and `text` hold the page's main content, without the menus, the share buttons or
|
|
43
|
+
the related posts. Send `onlyMainContent: false` for the whole page, or pick the parts
|
|
44
|
+
yourself with CSS selectors:
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
const post = await scrapa.scrape({ url: "https://example.com/blog/hello", includeTags: ["article"] });
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
A link to a PDF or a Word file comes back as text too, and `metadata.pages` says how many
|
|
51
|
+
pages the PDF has.
|
|
52
|
+
|
|
42
53
|
Ask for a screenshot or a PDF and the browser is used whatever `engine` says. Both come back
|
|
43
54
|
base64-encoded, and asking for both costs no more than asking for one.
|
|
44
55
|
|
|
@@ -76,7 +87,7 @@ takes the default country and leaves a default city out.
|
|
|
76
87
|
## Searching the web
|
|
77
88
|
|
|
78
89
|
`search()` runs a real search and hands back results we read ourselves. That is not a
|
|
79
|
-
detail
|
|
90
|
+
detail: every search vendor forbids redistributing their results through another API, so
|
|
80
91
|
our own fetch is the only kind you are allowed to build on.
|
|
81
92
|
|
|
82
93
|
```ts
|
|
@@ -86,7 +97,7 @@ const found = await scrapa.search({
|
|
|
86
97
|
proxy: { country: "br" }, // picks the exit AND the market: Brazilian results, in Portuguese
|
|
87
98
|
});
|
|
88
99
|
|
|
89
|
-
found.results[0].url; // array order IS the ranking
|
|
100
|
+
found.results[0].url; // array order IS the ranking: there is no score to sort by, because none is real
|
|
90
101
|
found.cached; // the same question inside the window costs nothing and says so
|
|
91
102
|
```
|
|
92
103
|
|
|
@@ -95,8 +106,27 @@ asked of the front *and* enforced on the answer, so what comes back is on your l
|
|
|
95
106
|
the front did with the operator. `answer: true` adds a paragraph written over the hits. To
|
|
96
107
|
read any result, hand its URL straight to `scrape()`.
|
|
97
108
|
|
|
98
|
-
A search every front refused costs nothing. A search that ran and found nothing is charged
|
|
99
|
-
|
|
109
|
+
A search every front refused costs nothing. A search that ran and found nothing is charged.
|
|
110
|
+
The index answered, and "nothing published that" is an answer.
|
|
111
|
+
|
|
112
|
+
## Listing a site's pages
|
|
113
|
+
|
|
114
|
+
`map()` hands back every URL a site lists, without reading the pages. The list comes from the
|
|
115
|
+
site's robots.txt, its sitemaps (gzipped ones and indexes too) and the links on the page you
|
|
116
|
+
name.
|
|
117
|
+
|
|
118
|
+
```ts
|
|
119
|
+
const { urls } = await scrapa.map({ url: "https://example.com" });
|
|
120
|
+
|
|
121
|
+
urls[0]; // { url: "https://example.com", lastmod: "2026-10-01" }
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Each URL comes back once, without `utm_` parameters, a `#` part or a trailing slash. Only the
|
|
125
|
+
host you named is kept unless you pass `includeSubdomains: true`, and `search: "blog"` keeps
|
|
126
|
+
the URLs that contain the word. `truncated: true` means the site may list more than you got.
|
|
127
|
+
|
|
128
|
+
The price does not change with what a map finds. Only managed-pool bandwidth past the first
|
|
129
|
+
megabyte is added. Price one first with `quoteMap()`.
|
|
100
130
|
|
|
101
131
|
## Driving a real browser
|
|
102
132
|
|
|
@@ -121,8 +151,8 @@ try {
|
|
|
121
151
|
```
|
|
122
152
|
|
|
123
153
|
`create()` returns only once Chromium is actually up, so `connectUrl` is a required field and
|
|
124
|
-
needs no `!`. It carries the session's token, is shown once and cannot be rebuilt from the id
|
|
125
|
-
|
|
154
|
+
needs no `!`. It carries the session's token, is shown once and cannot be rebuilt from the id.
|
|
155
|
+
That is why `get()` does not repeat it.
|
|
126
156
|
|
|
127
157
|
Minutes are reserved for the whole window and settled by what you used, so closing early
|
|
128
158
|
hands the rest back. A browser that never came up costs nothing at all, and if you forget to
|
|
@@ -131,7 +161,7 @@ close one, it is reaped at `expiresAt` and settled the same way.
|
|
|
131
161
|
## Reading a list of pages
|
|
132
162
|
|
|
133
163
|
A batch is N scrapes on our schedule instead of your connection. Same options, same engines,
|
|
134
|
-
same meter
|
|
164
|
+
same meter. What you buy is not holding a socket open for an hour.
|
|
135
165
|
|
|
136
166
|
```ts
|
|
137
167
|
const batch = await scrapa.batches.create({
|
|
@@ -156,7 +186,7 @@ for (let from = 0; from < status.total; ) {
|
|
|
156
186
|
|
|
157
187
|
Nothing is charged when you submit and nothing is reserved: each URL then costs what the same
|
|
158
188
|
call would have cost alone, and `cost` on the batch is what it has spent so far. Because
|
|
159
|
-
nothing is held up front, a batch can outlive the credits that started it
|
|
189
|
+
nothing is held up front, a batch can outlive the credits that started it. Those URLs come
|
|
160
190
|
back `skipped` rather than `failed`, which is not a verdict about the URL, and you can submit
|
|
161
191
|
exactly those positions again.
|
|
162
192
|
|
|
@@ -165,14 +195,14 @@ exactly those positions again.
|
|
|
165
195
|
hundred items per read.
|
|
166
196
|
|
|
167
197
|
Rather not poll? Pass a `webhook` and we will POST when it is done. The signing secret comes
|
|
168
|
-
back once, on the create
|
|
198
|
+
back once, on the create. Verify a delivery with HMAC-SHA256 over `<timestamp>.<raw body>`
|
|
169
199
|
against the `v1=` value in the `scrapa-signature` header, and refuse a `t=` more than five
|
|
170
200
|
minutes old, or a delivery somebody copied can be sent to you again.
|
|
171
201
|
|
|
172
202
|
## What it costs, per call
|
|
173
203
|
|
|
174
204
|
Every metered answer carries what it cost and what is left. Methods hand you the page itself,
|
|
175
|
-
so the meter arrives through a callback
|
|
205
|
+
so the meter arrives through a callback, per call, rather than a mutable field two concurrent
|
|
176
206
|
scrapes would fight over.
|
|
177
207
|
|
|
178
208
|
```ts
|
|
@@ -200,11 +230,11 @@ try {
|
|
|
200
230
|
} catch (error) {
|
|
201
231
|
if (!(error instanceof ScrapaError)) throw error;
|
|
202
232
|
switch (error.outcome) {
|
|
203
|
-
case "blocked": // the site said no
|
|
233
|
+
case "blocked": // the site said no: try another pool, country or engine
|
|
204
234
|
return scrapa.scrape({ url, proxy: { pool: "residential", country: "de" } });
|
|
205
|
-
case "denied": // our policy said no
|
|
235
|
+
case "denied": // our policy said no: retrying cannot help, change the URL
|
|
206
236
|
return null;
|
|
207
|
-
case "broken": // the API said it broke, and refunded the call
|
|
237
|
+
case "broken": // the API said it broke, and refunded the call. Worth one retry
|
|
208
238
|
return scrapa.scrape({ url });
|
|
209
239
|
}
|
|
210
240
|
throw error; // your own mistake (a bad field, a missing key, a spent quota), or error.mayHaveRun
|
package/dist/client.js
CHANGED
|
@@ -53,9 +53,9 @@ export class Scrapa extends GeneratedOperations {
|
|
|
53
53
|
super();
|
|
54
54
|
const opts = typeof options === "string" ? { apiKey: options } : options;
|
|
55
55
|
if (!opts.apiKey)
|
|
56
|
-
throw new TypeError("scrapa: an API key is required
|
|
56
|
+
throw new TypeError("scrapa: an API key is required. Create one at https://console.scrapa.app/keys");
|
|
57
57
|
if (typeof (opts.fetch ?? globalThis.fetch) !== "function")
|
|
58
|
-
throw new TypeError("scrapa: no global fetch
|
|
58
|
+
throw new TypeError("scrapa: no global fetch. Use Node 20+, or pass `fetch` in the options.");
|
|
59
59
|
const baseUrl = opts.baseUrl ?? DEFAULT_BASE_URL;
|
|
60
60
|
const host = new URL(baseUrl).host;
|
|
61
61
|
const timeout = opts.timeout ?? 65_000;
|
package/dist/client.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"client.js","sourceRoot":"","sources":["../src/client.ts"],"names":[],"mappings":"AAAA,8FAA8F;AAC9F,6FAA6F;AAC7F,EAAE;AACF,0FAA0F;AAC1F,2FAA2F;AAC3F,4FAA4F;AAC5F,8FAA8F;AAC9F,8FAA8F;AAC9F,4FAA4F;AAC5F,2FAA2F;AAC3F,WAAW;AACX,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,OAAO,EAAkC,MAAM,yBAAyB,CAAC;AAClF,OAAO,EAAE,gBAAgB,EAAE,mBAAmB,EAAE,WAAW,EAAE,MAAM,2BAA2B,CAAC;AAC/F,OAAO,EACH,IAAI,GAIP,MAAM,0BAA0B,CAAC;AAMlC;;8BAE8B;AAC9B,MAAM,cAAc,GAAG,aAAa,CAAC;AAgDrC;;;;;;;GAOG;AACH,MAAM,UAAU,iBAAiB,CAC7B,MAAc,EACd,QAAmC,EACnC,OAAsC;IAEtC,IAAI,CAAC,QAAQ,IAAI,CAAC,OAAO;QAAE,OAAO,MAAM,CAAC;IACzC,MAAM,GAAG,GACL,OAAO,IAAI,MAAM,IAAI,OAAO,MAAM,CAAC,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,KAAK,KAAK,IAAI;QAC1E,CAAC,CAAC,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE;QACrB,CAAC,CAAC,EAAE,CAAC;IACb,MAAM,MAAM,GAA4B,EAAE,CAAC;IAC3C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC;QAC/C,IAAI,KAAK,KAAK,SAAS,IAAI,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAC;YAAE,MAAM,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;IAC1E,IAAI,GAAG,CAAC,IAAI,KAAK,SAAS,IAAI,GAAG,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QACrD,OAAO,MAAM,CAAC,IAAI,CAAC;QACnB,OAAO,MAAM,CAAC,MAAM,CAAC;IACzB,CAAC;IACD,IAAI,GAAG,CAAC,OAAO,KAAK,SAAS,EAAE,CAAC;QAC5B,OAAO,MAAM,CAAC,KAAK,CAAC;QACpB,OAAO,MAAM,CAAC,IAAI,CAAC;IACvB,CAAC;IACD,MAAM,KAAK,GAAG,EAAE,GAAG,MAAM,EAAE,GAAG,GAAG,EAAE,CAAC;IACpC,OAAO,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,GAAG,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;AACrE,CAAC;AAED,MAAM,OAAO,MAAO,SAAQ,mBAAmB;IAC1B,SAAS,CAAY;IACrB,aAAa,CAA4B;IAE1D,YAAY,OAA+B;QACvC,KAAK,EAAE,CAAC;QACR,MAAM,IAAI,GAAG,OAAO,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC;QACzE,IAAI,CAAC,IAAI,CAAC,MAAM;YACZ,MAAM,IAAI,SAAS,CACf,
|
|
1
|
+
{"version":3,"file":"client.js","sourceRoot":"","sources":["../src/client.ts"],"names":[],"mappings":"AAAA,8FAA8F;AAC9F,6FAA6F;AAC7F,EAAE;AACF,0FAA0F;AAC1F,2FAA2F;AAC3F,4FAA4F;AAC5F,8FAA8F;AAC9F,8FAA8F;AAC9F,4FAA4F;AAC5F,2FAA2F;AAC3F,WAAW;AACX,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,OAAO,EAAkC,MAAM,yBAAyB,CAAC;AAClF,OAAO,EAAE,gBAAgB,EAAE,mBAAmB,EAAE,WAAW,EAAE,MAAM,2BAA2B,CAAC;AAC/F,OAAO,EACH,IAAI,GAIP,MAAM,0BAA0B,CAAC;AAMlC;;8BAE8B;AAC9B,MAAM,cAAc,GAAG,aAAa,CAAC;AAgDrC;;;;;;;GAOG;AACH,MAAM,UAAU,iBAAiB,CAC7B,MAAc,EACd,QAAmC,EACnC,OAAsC;IAEtC,IAAI,CAAC,QAAQ,IAAI,CAAC,OAAO;QAAE,OAAO,MAAM,CAAC;IACzC,MAAM,GAAG,GACL,OAAO,IAAI,MAAM,IAAI,OAAO,MAAM,CAAC,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,KAAK,KAAK,IAAI;QAC1E,CAAC,CAAC,EAAE,GAAG,MAAM,CAAC,KAAK,EAAE;QACrB,CAAC,CAAC,EAAE,CAAC;IACb,MAAM,MAAM,GAA4B,EAAE,CAAC;IAC3C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC;QAC/C,IAAI,KAAK,KAAK,SAAS,IAAI,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAC;YAAE,MAAM,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;IAC1E,IAAI,GAAG,CAAC,IAAI,KAAK,SAAS,IAAI,GAAG,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QACrD,OAAO,MAAM,CAAC,IAAI,CAAC;QACnB,OAAO,MAAM,CAAC,MAAM,CAAC;IACzB,CAAC;IACD,IAAI,GAAG,CAAC,OAAO,KAAK,SAAS,EAAE,CAAC;QAC5B,OAAO,MAAM,CAAC,KAAK,CAAC;QACpB,OAAO,MAAM,CAAC,IAAI,CAAC;IACvB,CAAC;IACD,MAAM,KAAK,GAAG,EAAE,GAAG,MAAM,EAAE,GAAG,GAAG,EAAE,CAAC;IACpC,OAAO,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,GAAG,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;AACrE,CAAC;AAED,MAAM,OAAO,MAAO,SAAQ,mBAAmB;IAC1B,SAAS,CAAY;IACrB,aAAa,CAA4B;IAE1D,YAAY,OAA+B;QACvC,KAAK,EAAE,CAAC;QACR,MAAM,IAAI,GAAG,OAAO,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC;QACzE,IAAI,CAAC,IAAI,CAAC,MAAM;YACZ,MAAM,IAAI,SAAS,CACf,+EAA+E,CAClF,CAAC;QACN,IAAI,OAAO,CAAC,IAAI,CAAC,KAAK,IAAI,UAAU,CAAC,KAAK,CAAC,KAAK,UAAU;YACtD,MAAM,IAAI,SAAS,CACf,wEAAwE,CAC3E,CAAC;QACN,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,gBAAgB,CAAC;QACjD,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC;QACnC,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,MAAM,CAAC;QACvC,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,OAAO,CAAC,IAAI,OAAO,GAAG,CAAC,IAAI,OAAO,GAAG,cAAc;YACrE,MAAM,IAAI,UAAU,CAChB,yEAAyE,cAAc,GAAG,CAC7F,CAAC;QACN,IAAI,CAAC,SAAS,GAAG;YACb,OAAO;YACP,OAAO,EAAE;gBACL,aAAa,EAAE,UAAU,IAAI,CAAC,MAAM,EAAE;gBACtC,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,aAAa,WAAW,EAAE;aAC/C;YACD,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,SAAS,EAAE,GAAG,EAAE,CAAC,OAAO;YACxB,UAAU,EAAE,IAAI,CAAC,UAAU,IAAI,CAAC;YAChC,KAAK,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,WAAW,CAAC,OAAO,EAAE,IAAI,CAAC;SACjD,CAAC;QACF,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC,QAAQ,EAAE,KAAK,CAAC;IAC9C,CAAC;IAES,KAAK,CAAC,OAAO,CACnB,IAAiB,EACjB,SAAiB,EAAE,EACnB,OAAuB,EAAE;QAEzB,MAAM,YAAY,GAAG,iBAAiB,CAAC,MAAM,EAAE,IAAI,CAAC,aAAa,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC;QAC/E,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,MAAM,IAAI,CAAe,IAAI,CAAC,SAAS,EAAE,IAAI,EAAE,YAAY,EAAE,IAAI,CAAC,CAAC;QAC1F,IAAI,IAAI,KAAK,SAAS;YAAE,IAAI,CAAC,OAAO,EAAE,CAAC,IAAI,CAAC,CAAC;QAC7C,OAAO,IAAI,CAAC;IAChB,CAAC;CACJ"}
|
package/dist/errors.js
CHANGED
|
@@ -94,7 +94,7 @@ export function scrapaError(failure, host) {
|
|
|
94
94
|
status: 0,
|
|
95
95
|
mayHaveRun: method !== "GET",
|
|
96
96
|
message: failure.timedOut
|
|
97
|
-
? `scrapa: no answer within ${failure.timeoutMs}ms
|
|
97
|
+
? `scrapa: no answer within ${failure.timeoutMs}ms. The page may still have been read and charged.`
|
|
98
98
|
: `scrapa: could not reach ${host}.`,
|
|
99
99
|
cause: failure.cause,
|
|
100
100
|
});
|
package/dist/errors.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,EAAE;AACF,wFAAwF;AACxF,2FAA2F;AAC3F,qFAAqF;AACrF,yFAAyF;AACzF,2FAA2F;AAC3F,OAAO,EAAE,OAAO,EAAE,UAAU,EAAkB,MAAM,yBAAyB,CAAC;AAC9E,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AA6BtD,MAAM,OAAO,WAAY,SAAQ,KAAK;IACzB,IAAI,CAA8B;IAClC,MAAM,CAAS;IACf,UAAU,CAAU;IACpB,MAAM,CAAmC;IACzC,OAAO,CAAW;IAClB,UAAU,CAAU;IACpB,SAAS,CAAU;IAC5B;;;;;OAKG;IACM,UAAU,CAAU;IACpB,IAAI,CAAU;IACd,gBAAgB,CAAU;IAEnC,YAAY,IAAqB;QAC7B,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,IAAI,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;QAClF,IAAI,CAAC,IAAI,GAAG,aAAa,CAAC;QAC1B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;QAC5B,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,IAAI,KAAK,CAAC;QAC3C,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,gBAAgB,GAAG,IAAI,CAAC,gBAAgB,CAAC;IAClD,CAAC;IAED;;;;;;;;;;;;OAYG;IACH,IAAI,OAAO;QACP,OAAO,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,IAAiB,CAAC,CAAC;IAC5E,CAAC;IAED;;uEAEmE;IACnE,IAAI,SAAS;QACT,IAAI,IAAI,CAAC,UAAU;YAAE,OAAO,KAAK,CAAC;QAClC,OAAO,CACH,IAAI,CAAC,IAAI,KAAK,eAAe;YAC7B,IAAI,CAAC,IAAI,KAAK,SAAS;YACvB,IAAI,CAAC,IAAI,KAAK,qBAAqB;YACnC,IAAI,CAAC,OAAO,KAAK,QAAQ,CAC5B,CAAC;IACN,CAAC;CACJ;AAED,SAAS,YAAY,CAAC,OAA4B,EAAE,IAAY;IAC5D,MAAM,GAAG,GAAG,OAAO,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;IAC/B,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IACxD,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,OAAO,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;AACtD,CAAC;AAED,oEAAoE;AACpE,SAAS,UAAU,CAAC,MAAc;IAC9B,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,qBAAqB,CAAC;IACjD,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,WAAW,CAAC;IACvC,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,cAAc,CAAC;IAC1C,OAAO,gBAAgB,CAAC;AAC5B,CAAC;AAED;;4CAE4C;AAC5C,MAAM,UAAU,WAAW,CAAC,OAAgB,EAAE,IAAY;IACtD,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;IAC1D,IAAI,MAAM,KAAK,CAAC;QACZ,OAAO,IAAI,WAAW,CAAC;YACnB,IAAI,EAAE,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,eAAe,CAA2B;YAChF,MAAM,EAAE,CAAC;YACT,UAAU,EAAE,MAAM,KAAK,KAAK;YAC5B,OAAO,EAAE,OAAO,CAAC,QAAQ;gBACrB,CAAC,CAAC,4BAA4B,OAAO,CAAC,SAAS,
|
|
1
|
+
{"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,EAAE;AACF,wFAAwF;AACxF,2FAA2F;AAC3F,qFAAqF;AACrF,yFAAyF;AACzF,2FAA2F;AAC3F,OAAO,EAAE,OAAO,EAAE,UAAU,EAAkB,MAAM,yBAAyB,CAAC;AAC9E,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AA6BtD,MAAM,OAAO,WAAY,SAAQ,KAAK;IACzB,IAAI,CAA8B;IAClC,MAAM,CAAS;IACf,UAAU,CAAU;IACpB,MAAM,CAAmC;IACzC,OAAO,CAAW;IAClB,UAAU,CAAU;IACpB,SAAS,CAAU;IAC5B;;;;;OAKG;IACM,UAAU,CAAU;IACpB,IAAI,CAAU;IACd,gBAAgB,CAAU;IAEnC,YAAY,IAAqB;QAC7B,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,IAAI,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC;QAClF,IAAI,CAAC,IAAI,GAAG,aAAa,CAAC;QAC1B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;QAC5B,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;QAClC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,SAAS,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,IAAI,KAAK,CAAC;QAC3C,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC;QACtB,IAAI,CAAC,gBAAgB,GAAG,IAAI,CAAC,gBAAgB,CAAC;IAClD,CAAC;IAED;;;;;;;;;;;;OAYG;IACH,IAAI,OAAO;QACP,OAAO,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC,IAAiB,CAAC,CAAC;IAC5E,CAAC;IAED;;uEAEmE;IACnE,IAAI,SAAS;QACT,IAAI,IAAI,CAAC,UAAU;YAAE,OAAO,KAAK,CAAC;QAClC,OAAO,CACH,IAAI,CAAC,IAAI,KAAK,eAAe;YAC7B,IAAI,CAAC,IAAI,KAAK,SAAS;YACvB,IAAI,CAAC,IAAI,KAAK,qBAAqB;YACnC,IAAI,CAAC,OAAO,KAAK,QAAQ,CAC5B,CAAC;IACN,CAAC;CACJ;AAED,SAAS,YAAY,CAAC,OAA4B,EAAE,IAAY;IAC5D,MAAM,GAAG,GAAG,OAAO,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;IAC/B,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IACxD,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IAC1B,OAAO,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;AACtD,CAAC;AAED,oEAAoE;AACpE,SAAS,UAAU,CAAC,MAAc;IAC9B,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,qBAAqB,CAAC;IACjD,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,WAAW,CAAC;IACvC,IAAI,MAAM,KAAK,GAAG;QAAE,OAAO,cAAc,CAAC;IAC1C,OAAO,gBAAgB,CAAC;AAC5B,CAAC;AAED;;4CAE4C;AAC5C,MAAM,UAAU,WAAW,CAAC,OAAgB,EAAE,IAAY;IACtD,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;IAC1D,IAAI,MAAM,KAAK,CAAC;QACZ,OAAO,IAAI,WAAW,CAAC;YACnB,IAAI,EAAE,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,eAAe,CAA2B;YAChF,MAAM,EAAE,CAAC;YACT,UAAU,EAAE,MAAM,KAAK,KAAK;YAC5B,OAAO,EAAE,OAAO,CAAC,QAAQ;gBACrB,CAAC,CAAC,4BAA4B,OAAO,CAAC,SAAS,oDAAoD;gBACnG,CAAC,CAAC,2BAA2B,IAAI,GAAG;YACxC,KAAK,EAAE,OAAO,CAAC,KAAK;SACvB,CAAC,CAAC;IACP,sFAAsF;IACtF,0FAA0F;IAC1F,wFAAwF;IACxF,uEAAuE;IACvE,IAAI,MAAM,GAAG,GAAG;QACZ,OAAO,IAAI,WAAW,CAAC;YACnB,IAAI,EAAE,kBAA4C;YAClD,MAAM;YACN,UAAU,EAAE,IAAI;YAChB,OAAO,EAAE,WAAW,MAAM,IAAI,IAAI,aAAa,MAAM,kHAAkH;YACvK,OAAO,EAAE,OAAO,CAAC,IAAI;YACrB,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,IAAI,EAAE,YAAY,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,IAAI,CAAC;YACjD,gBAAgB,EAAE,YAAY,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,gBAAgB,CAAC;SAC5E,CAAC,CAAC;IACP,yFAAyF;IACzF,sFAAsF;IACtF,mFAAmF;IACnF,MAAM,UAAU,GAAG,CAAC,QAAQ,IAAI,MAAM,IAAI,GAAG,IAAI,MAAM,KAAK,KAAK,CAAC;IAClE,OAAO,IAAI,WAAW,CAAC;QACnB,IAAI,EAAE,QAAQ,CAAC,CAAC,CAAE,QAAQ,CAAC,IAAkB,CAAC,CAAC,CAAC,UAAU,CAAC,MAAM,CAAC;QAClE,MAAM;QACN,UAAU;QACV,mFAAmF;QACnF,gCAAgC;QAChC,OAAO,EACH,QAAQ,EAAE,OAAO;YACjB,WAAW,MAAM,IAAI,IAAI,gBAAgB,MAAM,IAAI,UAAU,CAAC,CAAC,CAAC,mFAAmF,CAAC,CAAC,CAAC,EAAE,EAAE;QAC9J,UAAU,EAAE,QAAQ,EAAE,UAAU;QAChC,MAAM,EAAE,QAAQ,EAAE,MAAM;QACxB,OAAO,EAAE,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI;QACnD,sEAAsE;QACtE,UAAU,EACN,cAAc,CAAC;YACX,cAAc,EAAE,OAAO,CAAC,cAAc;YACtC,OAAO,EAAE,QAAQ,EAAE,OAAO;SAC7B,CAAC,IAAI,SAAS;QACnB,SAAS,EAAE,OAAO,CAAC,SAAS;QAC5B,uFAAuF;QACvF,mFAAmF;QACnF,IAAI,EAAE,YAAY,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,IAAI,CAAC;QACjD,gBAAgB,EAAE,YAAY,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC,gBAAgB,CAAC;KAC5E,CAAC,CAAC;AACP,CAAC"}
|
|
@@ -4,12 +4,13 @@
|
|
|
4
4
|
* locale catalogs, exhaustiveness checks).
|
|
5
5
|
*
|
|
6
6
|
* THE WHOSE-FAULT VOCABULARY (DESIGN.md) is the organising principle: a caller's next move
|
|
7
|
-
* depends entirely on who said no, so the code says. `UPSTREAM_BLOCKED`
|
|
8
|
-
*
|
|
7
|
+
* depends entirely on who said no, so the code says. `UPSTREAM_BLOCKED` is "the site said no"
|
|
8
|
+
* (retry differently — another engine, pool or country), and `NAV_TIMEOUT` shares its hue under
|
|
9
|
+
* its own word, "time ran out" (give it more time; see `OWN_WORD`); `SSRF_DENIED` /
|
|
9
10
|
* `AUP_DENIED` are "our policy said no" (retrying cannot help, change the URL); a 5xx is
|
|
10
11
|
* "we broke it" (retry as-is, already refunded).
|
|
11
12
|
*/
|
|
12
|
-
export declare const ERROR_CODES: readonly ["VALIDATION_ERROR", "INVALID_URL", "SSRF_DENIED", "UNAUTHORIZED", "INVALID_TOKEN", "PLAN_GATE", "FORBIDDEN", "AUP_DENIED", "NOT_FOUND", "SESSION_NOT_FOUND", "CONFLICT", "PAYLOAD_TOO_LARGE", "UPSTREAM_BLOCKED", "NAV_TIMEOUT", "PROXY_REFUSED", "RATE_LIMIT_EXCEEDED", "QUOTA_EXCEEDED", "INTERNAL_ERROR", "SERVICE_UNAVAILABLE", "BILLING_UNAVAILABLE"];
|
|
13
|
+
export declare const ERROR_CODES: readonly ["VALIDATION_ERROR", "INVALID_URL", "SSRF_DENIED", "UNAUTHORIZED", "INVALID_TOKEN", "PLAN_GATE", "FORBIDDEN", "AUP_DENIED", "NOT_FOUND", "SESSION_NOT_FOUND", "CONFLICT", "PAYLOAD_TOO_LARGE", "UPSTREAM_BLOCKED", "NAV_TIMEOUT", "PROXY_REFUSED", "FILE_TOO_LARGE", "DOCUMENT_TIMEOUT", "RATE_LIMIT_EXCEEDED", "QUOTA_EXCEEDED", "INTERNAL_ERROR", "SERVICE_UNAVAILABLE", "BILLING_UNAVAILABLE"];
|
|
13
14
|
export type ErrorCode = (typeof ERROR_CODES)[number];
|
|
14
15
|
/** The status each code answers with. The server's error factory, the SDK's fallback and
|
|
15
16
|
* the docs' errors table all read this one map — three copies could drift, one cannot. */
|
|
@@ -29,6 +30,8 @@ export declare const ERROR_STATUS: {
|
|
|
29
30
|
readonly UPSTREAM_BLOCKED: 422;
|
|
30
31
|
readonly NAV_TIMEOUT: 422;
|
|
31
32
|
readonly PROXY_REFUSED: 422;
|
|
33
|
+
readonly FILE_TOO_LARGE: 422;
|
|
34
|
+
readonly DOCUMENT_TIMEOUT: 422;
|
|
32
35
|
readonly RATE_LIMIT_EXCEEDED: 429;
|
|
33
36
|
readonly QUOTA_EXCEEDED: 429;
|
|
34
37
|
readonly INTERNAL_ERROR: 500;
|
|
@@ -36,10 +39,11 @@ export declare const ERROR_STATUS: {
|
|
|
36
39
|
readonly BILLING_UNAVAILABLE: 503;
|
|
37
40
|
};
|
|
38
41
|
/**
|
|
39
|
-
* The three no-charge outcomes, as DESIGN.md names them.
|
|
40
|
-
*
|
|
41
|
-
*
|
|
42
|
-
*
|
|
42
|
+
* The three no-charge outcomes, as DESIGN.md names them. Each code here maps to exactly one
|
|
43
|
+
* — this is the map the console reads to pick the hue and the word, and the SDK reads to
|
|
44
|
+
* tell a caller what to do next. A code absent from here gets no colour: the caller's own
|
|
45
|
+
* mistake (a bad body, a missing key), or a free refusal whose fix is in the request rather
|
|
46
|
+
* than the site (PROXY_REFUSED, FILE_TOO_LARGE, DOCUMENT_TIMEOUT).
|
|
43
47
|
*
|
|
44
48
|
* blocked → the site said no → try another engine, pool or country
|
|
45
49
|
* denied → our policy said no → change the URL; retrying cannot help
|
|
@@ -115,8 +119,9 @@ export type ResolvedEngine = Exclude<Engine, "auto">;
|
|
|
115
119
|
/** What a scrape can hand back. `markdown` is the default because it is what an LLM, a
|
|
116
120
|
* notebook and a human all read; `html` is the page as fetched; `text` is prose with the
|
|
117
121
|
* markup gone; `links` every absolute anchor in document order; `extract` is the page read
|
|
118
|
-
* into the fields the caller asked for, which is the one format a model writes
|
|
119
|
-
|
|
122
|
+
* into the fields the caller asked for, which is the one format a model writes; `raw` is the
|
|
123
|
+
* response's own bytes, the file a browser would save. */
|
|
124
|
+
export declare const SCRAPE_FORMATS: readonly ["markdown", "html", "text", "links", "screenshot", "pdf", "extract", "raw"];
|
|
120
125
|
export type ScrapeFormat = (typeof SCRAPE_FORMATS)[number];
|
|
121
126
|
/** A session's life. `starting` is the driver bringing Chromium up; `live` is the only
|
|
122
127
|
* status whose `connectUrl` accepts a CDP client. The three end states answer one
|
|
@@ -131,8 +136,10 @@ export declare const SEARCH_TOPICS: readonly ["general", "news"];
|
|
|
131
136
|
export type SearchTopic = (typeof SEARCH_TOPICS)[number];
|
|
132
137
|
/** Which front answered a search. Not a caller's choice — the ladder picks, and says which
|
|
133
138
|
* one it was, because a result set's flavour follows its index. */
|
|
134
|
-
export declare const SEARCH_ENGINES: readonly ["bing", "brave"];
|
|
139
|
+
export declare const SEARCH_ENGINES: readonly ["bing", "brave", "duckduckgo"];
|
|
135
140
|
export type SearchEngine = (typeof SEARCH_ENGINES)[number];
|
|
141
|
+
/** Where a map's URLs came from. With none found, the sources that were read. */
|
|
142
|
+
export type MapSource = "sitemap" | "links" | "both";
|
|
136
143
|
/** A batch's life. `queued` is accepted but not yet picked up; `running` has at least one
|
|
137
144
|
* item in flight. The three end states answer the same question the session statuses do —
|
|
138
145
|
* who ended it: `completed` ran every item to a verdict (some of which may be failures,
|
|
@@ -183,6 +190,19 @@ export interface ScrapeMetadataDto {
|
|
|
183
190
|
* it so a caller can retry with `engine: "browser"` instead of trusting an empty page.
|
|
184
191
|
* Always false from the browser engine, which ran the scripts. */
|
|
185
192
|
needsJs: boolean;
|
|
193
|
+
/** How many pages the PDF has, when a PDF was read into text. Absent for anything else,
|
|
194
|
+
* and when only "raw" was asked for. */
|
|
195
|
+
pages?: number;
|
|
196
|
+
/** True when the PDF has more pages than we read: the first 200 are in the answer and the
|
|
197
|
+
* rest are not. Absent otherwise. */
|
|
198
|
+
truncated?: boolean;
|
|
199
|
+
}
|
|
200
|
+
/** A file, exactly as the site sent it. */
|
|
201
|
+
export interface ScrapeRawDto {
|
|
202
|
+
/** The bytes, base64-encoded. */
|
|
203
|
+
base64: string;
|
|
204
|
+
/** The response's `content-type`, media type only: `application/pdf`. */
|
|
205
|
+
contentType: string | null;
|
|
186
206
|
}
|
|
187
207
|
/** `POST /v1/scrape`'s answer. Only the requested formats are present; everything else is
|
|
188
208
|
* always there. Artifacts (`screenshot`, `pdf`) are base64 in v1. */
|
|
@@ -214,6 +234,9 @@ export interface ScrapeResultDto {
|
|
|
214
234
|
links?: string[];
|
|
215
235
|
screenshot?: string;
|
|
216
236
|
pdf?: string;
|
|
237
|
+
/** The response's own bytes, as a browser would save them (any compression undone), when
|
|
238
|
+
* `raw` was in `formats`. Never cut short: a file over the limit is refused instead. */
|
|
239
|
+
raw?: ScrapeRawDto;
|
|
217
240
|
/** The fields asked for in `extract`, read off the page by a model. Null when the model
|
|
218
241
|
* could not answer — that credit is handed back and the page still stands. Present only
|
|
219
242
|
* when `extract` was in `formats`. */
|
|
@@ -352,6 +375,39 @@ export interface CreatedBatchDto extends BatchDto {
|
|
|
352
375
|
* delivery somebody copied can be sent to you again. */
|
|
353
376
|
webhookSecret?: string;
|
|
354
377
|
}
|
|
378
|
+
/** One URL a site lists, normalised: no fragment, no `utm_*` parameters, the query sorted,
|
|
379
|
+
* no trailing slash. */
|
|
380
|
+
export interface MapUrlDto {
|
|
381
|
+
url: string;
|
|
382
|
+
/** When the sitemap says the page last changed, as the sitemap wrote it (a W3C date, often
|
|
383
|
+
* just the day). Absent when it did not say, or when the URL came from the page's links. */
|
|
384
|
+
lastmod?: string;
|
|
385
|
+
}
|
|
386
|
+
/** `POST /v1/map`'s answer. */
|
|
387
|
+
export interface MapResultDto {
|
|
388
|
+
/** Every URL found, once each, sitemap URLs first in the order the site listed them. */
|
|
389
|
+
urls: MapUrlDto[];
|
|
390
|
+
/** Where they came from. With none found, the sources that were read. */
|
|
391
|
+
source: MapSource;
|
|
392
|
+
/** True when the site may list more than this answer holds: `limit` cut the list, or the
|
|
393
|
+
* robots.txt or a sitemap was not read to its end (the site refused it, its size, the
|
|
394
|
+
* 25-document cap, or the time limit). */
|
|
395
|
+
truncated: boolean;
|
|
396
|
+
pool: ProxyPool;
|
|
397
|
+
/** What the managed pool carried for every document the map read, decompressed. The
|
|
398
|
+
* first megabyte is free; past it, bandwidth is metered like any request's. */
|
|
399
|
+
bytes: number;
|
|
400
|
+
}
|
|
401
|
+
/** What a map would cost, without making it. */
|
|
402
|
+
export interface MapQuoteDto {
|
|
403
|
+
pool: ProxyPool;
|
|
404
|
+
/** Credits a map charges, whatever it finds. */
|
|
405
|
+
cost: number;
|
|
406
|
+
/** Managed-pool megabytes included before `perMb` applies. 0 on BYO. */
|
|
407
|
+
freeMb: number;
|
|
408
|
+
/** Credits per additional megabyte through the managed pool. 0 on BYO. */
|
|
409
|
+
perMb: number;
|
|
410
|
+
}
|
|
355
411
|
/** One hit, as the index ranked it. The array's ORDER is the ranking — there is no score,
|
|
356
412
|
* because no engine we read publishes one and a rank-derived float would be a made-up
|
|
357
413
|
* number in a public contract. */
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"contract.d.ts","sourceRoot":"","sources":["../../src/generated/contract.ts"],"names":[],"mappings":"AAOA
|
|
1
|
+
{"version":3,"file":"contract.d.ts","sourceRoot":"","sources":["../../src/generated/contract.ts"],"names":[],"mappings":"AAOA;;;;;;;;;;;GAWG;AACH,eAAO,MAAM,WAAW,4YA0Cd,CAAC;AAEX,MAAM,MAAM,SAAS,GAAG,CAAC,OAAO,WAAW,CAAC,CAAC,MAAM,CAAC,CAAC;AAErD;2FAC2F;AAC3F,eAAO,MAAM,YAAY;;;;;;;;;;;;;;;;;;;;;;;CAuBqB,CAAC;AAE/C;;;;;;;;;;GAUG;AACH,eAAO,MAAM,UAAU,EAAE,OAAO,CAAC,MAAM,CAAC,SAAS,EAAE,SAAS,GAAG,QAAQ,GAAG,QAAQ,CAAC,CAUlF,CAAC;AAEF;;;;;;;GAOG;AACH,eAAO,MAAM,OAAO;IAChB,uEAAuE;;IAEvE,+EAA+E;;;IAG/E,sEAAsE;;IAEtE,yCAAyC;;;CAGnC,CAAC;AAEX,MAAM,WAAW,cAAc;IAC3B,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,OAAO,CAAC;CACpB;AAED;;kEAEkE;AAClE,MAAM,WAAW,SAAS;IACtB,+EAA+E;IAC/E,IAAI,EAAE,MAAM,CAAC;IACb,gBAAgB,EAAE,MAAM,CAAC;IACzB,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,CAAC,EAAE,IAAI,CAAC;CACjB;AAED,oEAAoE;AACpE,MAAM,WAAW,UAAU,CAAC,CAAC;IACzB,IAAI,EAAE,CAAC,CAAC;IACR,IAAI,CAAC,EAAE,cAAc,GAAG,SAAS,CAAC;CACrC;AAED,uFAAuF;AACvF,MAAM,WAAW,QAAQ;IACrB,KAAK,EAAE;QACH,IAAI,EAAE,SAAS,CAAC;QAChB,yFAAyF;QACzF,OAAO,EAAE,MAAM,CAAC;QAChB,gFAAgF;QAChF,UAAU,CAAC,EAAE,MAAM,CAAC;QACpB,6CAA6C;QAC7C,MAAM,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC,CAAC;QACzC,OAAO,CAAC,EAAE,OAAO,CAAC;KACrB,CAAC;CACL;AAGD;;;wDAGwD;AACxD,eAAO,MAAM,WAAW,oCAAqC,CAAC;AAE9D,MAAM,MAAM,SAAS,GAAG,CAAC,OAAO,WAAW,CAAC,CAAC,MAAM,CAAC,CAAC;AAErD;iFACiF;AACjF,eAAO,MAAM,OAAO,uCAAwC,CAAC;AAE7D,MAAM,MAAM,MAAM,GAAG,CAAC,OAAO,OAAO,CAAC,CAAC,MAAM,CAAC,CAAC;AAE9C,4FAA4F;AAC5F,MAAM,MAAM,cAAc,GAAG,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;AAErD;;;;2DAI2D;AAC3D,eAAO,MAAM,cAAc,uFASjB,CAAC;AAEX,MAAM,MAAM,YAAY,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,CAAC,CAAC;AAE3D;;;;4EAI4E;AAC5E,eAAO,MAAM,gBAAgB,8DAA+D,CAAC;AAE7F,MAAM,MAAM,aAAa,GAAG,CAAC,OAAO,gBAAgB,CAAC,CAAC,MAAM,CAAC,CAAC;AAE9D;qFACqF;AACrF,eAAO,MAAM,aAAa,8BAA+B,CAAC;AAE1D,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,aAAa,CAAC,CAAC,MAAM,CAAC,CAAC;AAEzD;oEACoE;AACpE,eAAO,MAAM,cAAc,0CAA2C,CAAC;AAEvE,MAAM,MAAM,YAAY,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,CAAC,CAAC;AAE3D,iFAAiF;AACjF,MAAM,MAAM,SAAS,GAAG,SAAS,GAAG,OAAO,GAAG,MAAM,CAAC;AAErD;;;;mEAImE;AACnE,eAAO,MAAM,cAAc,mEAAoE,CAAC;AAEhG,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,CAAC,CAAC;AAE1D;;;;;eAKe;AACf,eAAO,MAAM,mBAAmB,6DAA8D,CAAC;AAE/F,MAAM,MAAM,eAAe,GAAG,CAAC,OAAO,mBAAmB,CAAC,CAAC,MAAM,CAAC,CAAC;AAEnE;;;;;;8BAM8B;AAC9B,eAAO,MAAM,cAAc,sDAAuD,CAAC;AAEnF,MAAM,MAAM,YAAY,GAAG,CAAC,OAAO,cAAc,CAAC,CAAC,MAAM,CAAC,CAAC;AAE3D;;;kBAGkB;AAClB,eAAO,MAAM,yBAAyB,6CAA8C,CAAC;AAErF,MAAM,MAAM,qBAAqB,GAAG,CAAC,OAAO,yBAAyB,CAAC,CAAC,MAAM,CAAC,CAAC;AAG/E,MAAM,MAAM,MAAM,GAAG,MAAM,GAAG,SAAS,GAAG,KAAK,GAAG,OAAO,GAAG,UAAU,CAAC;AAGvE;qEACqE;AACrE,MAAM,WAAW,iBAAiB;IAC9B,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3B,yDAAyD;IACzD,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,aAAa,EAAE,MAAM,GAAG,IAAI,CAAC;IAC7B,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,sDAAsD;IACtD,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3B;;;uEAGmE;IACnE,OAAO,EAAE,OAAO,CAAC;IACjB;6CACyC;IACzC,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;0CACsC;IACtC,SAAS,CAAC,EAAE,OAAO,CAAC;CACvB;AAED,2CAA2C;AAC3C,MAAM,WAAW,YAAY;IACzB,iCAAiC;IACjC,MAAM,EAAE,MAAM,CAAC;IACf,yEAAyE;IACzE,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;CAC9B;AAED;sEACsE;AACtE,MAAM,WAAW,eAAe;IAC5B,4BAA4B;IAC5B,GAAG,EAAE,MAAM,CAAC;IACZ,oDAAoD;IACpD,QAAQ,EAAE,MAAM,CAAC;IACjB,wFAAwF;IACxF,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,cAAc,CAAC;IACvB,IAAI,EAAE,SAAS,CAAC;IAChB;;;mDAG+C;IAC/C,SAAS,CAAC,EAAE,OAAO,CAAC;IACpB;;gDAE4C;IAC5C,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB;;8FAE0F;IAC1F,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,EAAE,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb;6FACyF;IACzF,GAAG,CAAC,EAAE,YAAY,CAAC;IACnB;;2CAEuC;IACvC,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,IAAI,CAAC;IACzC,QAAQ,EAAE,iBAAiB,CAAC;CAC/B;AAED,0EAA0E;AAC1E,MAAM,WAAW,cAAc;IAC3B,MAAM,EAAE,cAAc,CAAC;IACvB,IAAI,EAAE,SAAS,CAAC;IAChB,OAAO,EAAE,YAAY,EAAE,CAAC;IACxB,mFAAmF;IACnF,IAAI,EAAE,MAAM,CAAC;IACb;sFACkF;IAClF,OAAO,EAAE,MAAM,CAAC;IAChB,oFAAoF;IACpF,MAAM,EAAE,MAAM,CAAC;IACf,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;CACjB;AAGD;;;;;;;;;;;;;;GAcG;AACH,MAAM,WAAW,QAAQ;IACrB,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,WAAW,CAAC;IACpB,4DAA4D;IAC5D,KAAK,EAAE,MAAM,CAAC;IACd;gDAC4C;IAC5C,SAAS,EAAE,MAAM,CAAC;IAClB;;uEAEmE;IACnE,MAAM,EAAE,MAAM,CAAC;IACf;;;mBAGe;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,8EAA8E;IAC9E,IAAI,EAAE,SAAS,CAAC;IAChB,wDAAwD;IACxD,IAAI,EAAE,MAAM,CAAC;IACb,8EAA8E;IAC9E,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,yCAAyC;IACzC,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;8DAG0D;IAC1D,SAAS,EAAE,MAAM,CAAC;IAClB;+DAC2D;IAC3D,KAAK,CAAC,EAAE,YAAY,EAAE,CAAC;IACvB,oFAAoF;IACpF,OAAO,CAAC,EAAE,eAAe,CAAC;CAC7B;AAED;uDACuD;AACvD,MAAM,WAAW,YAAY;IACzB;0EACsE;IACtE,QAAQ,EAAE,MAAM,CAAC;IACjB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,eAAe,CAAC;IACxB,oEAAoE;IACpE,MAAM,CAAC,EAAE,eAAe,CAAC;IACzB;0DACsD;IACtD,KAAK,CAAC,EAAE;QAAE,IAAI,EAAE,SAAS,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC;IAC7C,8EAA8E;IAC9E,IAAI,EAAE,MAAM,CAAC;CAChB;AAED,6CAA6C;AAC7C,MAAM,WAAW,eAAe;IAC5B,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,YAAY,EAAE,CAAC;IACvB;uEACmE;IACnE,MAAM,EAAE,qBAAqB,CAAC;IAC9B;8EAC0E;IAC1E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;yFAEqF;IACrF,QAAQ,EAAE,MAAM,CAAC;CACpB;AAED;;yCAEyC;AACzC,MAAM,WAAW,aAAa;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,IAAI,EAAE,SAAS,CAAC;IAChB;0CACsC;IACtC,cAAc,EAAE,MAAM,CAAC;IACvB;gCAC4B;IAC5B,OAAO,EAAE,MAAM,CAAC;CACnB;AAED;;;;;;;;GAQG;AACH,MAAM,WAAW,eAAgB,SAAQ,QAAQ;IAC7C;;;6DAGyD;IACzD,aAAa,CAAC,EAAE,MAAM,CAAC;CAC1B;AAGD;yBACyB;AACzB,MAAM,WAAW,SAAS;IACtB,GAAG,EAAE,MAAM,CAAC;IACZ;iGAC6F;IAC7F,OAAO,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,+BAA+B;AAC/B,MAAM,WAAW,YAAY;IACzB,wFAAwF;IACxF,IAAI,EAAE,SAAS,EAAE,CAAC;IAClB,yEAAyE;IACzE,MAAM,EAAE,SAAS,CAAC;IAClB;;+CAE2C;IAC3C,SAAS,EAAE,OAAO,CAAC;IACnB,IAAI,EAAE,SAAS,CAAC;IAChB;oFACgF;IAChF,KAAK,EAAE,MAAM,CAAC;CACjB;AAED,gDAAgD;AAChD,MAAM,WAAW,WAAW;IACxB,IAAI,EAAE,SAAS,CAAC;IAChB,gDAAgD;IAChD,IAAI,EAAE,MAAM,CAAC;IACb,wEAAwE;IACxE,MAAM,EAAE,MAAM,CAAC;IACf,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;CACjB;AAGD;;mCAEmC;AACnC,MAAM,WAAW,YAAY;IACzB,uCAAuC;IACvC,QAAQ,EAAE,MAAM,CAAC;IACjB,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,MAAM,CAAC;IACd;uDACmD;IACnD,OAAO,EAAE,MAAM,CAAC;IAChB;sEACkE;IAClE,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;CAC9B;AAED,kCAAkC;AAClC,MAAM,WAAW,eAAe;IAC5B,sEAAsE;IACtE,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,WAAW,CAAC;IACnB,4BAA4B;IAC5B,MAAM,EAAE,YAAY,CAAC;IACrB,IAAI,EAAE,SAAS,CAAC;IAChB;yEACqE;IACrE,KAAK,EAAE,MAAM,CAAC;IACd;sCACkC;IAClC,MAAM,EAAE,OAAO,CAAC;IAChB;4FACwF;IACxF,MAAM,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,OAAO,EAAE,YAAY,EAAE,CAAC;CAC3B;AAGD;;;;;GAKG;AACH,MAAM,WAAW,UAAU;IACvB,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,aAAa,CAAC;IACtB;;;kEAG8D;IAC9D,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB;4FACwF;IACxF,IAAI,EAAE,SAAS,CAAC;IAChB,oEAAoE;IACpE,UAAU,EAAE,MAAM,CAAC;IACnB,0DAA0D;IAC1D,SAAS,EAAE,MAAM,CAAC;IAClB,oFAAoF;IACpF,eAAe,EAAE,MAAM,CAAC;IACxB;uFACmF;IACnF,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,kFAAkF;IAClF,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,mEAAmE;IACnE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;CACpB;AAED;;;;;GAKG;AACH,MAAM,WAAW,cAAe,SAAQ,UAAU;IAC9C,UAAU,EAAE,MAAM,CAAC;CACtB;AAED;;8CAE8C;AAC9C,MAAM,WAAW,eAAe;IAC5B,IAAI,EAAE,SAAS,CAAC;IAChB,UAAU,EAAE,MAAM,CAAC;IACnB,+BAA+B;IAC/B,SAAS,EAAE,MAAM,CAAC;IAClB,yFAAyF;IACzF,OAAO,EAAE,MAAM,CAAC;IAChB,wEAAwE;IACxE,MAAM,EAAE,MAAM,CAAC;IACf,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;CACjB;AAGD;;kCAEkC;AAClC,MAAM,WAAW,WAAW;IACxB,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,OAAO,CAAC;IACjB,2DAA2D;IAC3D,OAAO,EAAE,MAAM,CAAC;IAChB;mEAC+D;IAC/D,KAAK,EAAE,MAAM,CAAC;CACjB;AAED,sFAAsF;AACtF,MAAM,WAAW,QAAQ;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,gDAAgD;IAChD,KAAK,EAAE,MAAM,CAAC;IACd,uDAAuD;IACvD,WAAW,EAAE,MAAM,CAAC;IACpB,gCAAgC;IAChC,eAAe,EAAE,MAAM,CAAC;IACxB;+FAC2F;IAC3F,aAAa,EAAE,MAAM,CAAC;IACtB,gFAAgF;IAChF,WAAW,EAAE,MAAM,CAAC;IACpB,eAAe,EAAE,MAAM,CAAC;IACxB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,IAAI,EAAE,WAAW,EAAE,CAAC;CACvB"}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// GENERATED FILE — do not edit. Run `bun run sdk:gen`.
|
|
2
|
-
// Source: packages/shared/src/http/index.ts, packages/shared/src/locale/index.ts, packages/shared/src/enums/index.ts, packages/shared/src/billing/catalog.ts, packages/shared/src/dto/account.ts, packages/shared/src/dto/scrape.ts, packages/shared/src/dto/batch.ts, packages/shared/src/dto/search.ts, packages/shared/src/dto/sessions.ts, packages/shared/src/dto/usage.ts
|
|
2
|
+
// Source: packages/shared/src/http/index.ts, packages/shared/src/locale/index.ts, packages/shared/src/enums/index.ts, packages/shared/src/billing/catalog.ts, packages/shared/src/dto/account.ts, packages/shared/src/dto/scrape.ts, packages/shared/src/dto/batch.ts, packages/shared/src/dto/map.ts, packages/shared/src/dto/search.ts, packages/shared/src/dto/sessions.ts, packages/shared/src/dto/usage.ts
|
|
3
3
|
//
|
|
4
4
|
// The shapes the API speaks, lifted from the server's own contract.
|
|
5
5
|
// ── from packages/shared/src/http/index.ts ───────────────────────────
|
|
@@ -9,8 +9,9 @@
|
|
|
9
9
|
* locale catalogs, exhaustiveness checks).
|
|
10
10
|
*
|
|
11
11
|
* THE WHOSE-FAULT VOCABULARY (DESIGN.md) is the organising principle: a caller's next move
|
|
12
|
-
* depends entirely on who said no, so the code says. `UPSTREAM_BLOCKED`
|
|
13
|
-
*
|
|
12
|
+
* depends entirely on who said no, so the code says. `UPSTREAM_BLOCKED` is "the site said no"
|
|
13
|
+
* (retry differently — another engine, pool or country), and `NAV_TIMEOUT` shares its hue under
|
|
14
|
+
* its own word, "time ran out" (give it more time; see `OWN_WORD`); `SSRF_DENIED` /
|
|
14
15
|
* `AUP_DENIED` are "our policy said no" (retrying cannot help, change the URL); a 5xx is
|
|
15
16
|
* "we broke it" (retry as-is, already refunded).
|
|
16
17
|
*/
|
|
@@ -42,6 +43,15 @@ export const ERROR_CODES = [
|
|
|
42
43
|
// or did not answer, or the managed pool has no exit in the place you asked for. Free,
|
|
43
44
|
// and no colour — the fix is in the request (the proxy, the credentials, a wider place).
|
|
44
45
|
"PROXY_REFUSED", // 422 — details {reason: "no_exit" | "refused" | "unreachable", place?, status?}
|
|
46
|
+
// The site sent a file bigger than `raw` hands back whole. Not 413, which tells an HTTP
|
|
47
|
+
// client its OWN request was too big. Free, and no colour, for PROXY_REFUSED's reason:
|
|
48
|
+
// the fix is in the request (another format, or download it yourself), not another road.
|
|
49
|
+
"FILE_TOO_LARGE", // 422 — details {limit}, in bytes
|
|
50
|
+
// A PDF or a Word file arrived whole, and reading it into text took longer than the time
|
|
51
|
+
// the call had left. Free, and no colour: the site answered in full, so another engine,
|
|
52
|
+
// pool or country reads the same file just as slowly. The fix is more `timeoutMs`, or
|
|
53
|
+
// "raw" and reading it yourself.
|
|
54
|
+
"DOCUMENT_TIMEOUT", // 422: details {budgetMs}, the call's timeout
|
|
45
55
|
"RATE_LIMIT_EXCEEDED", // 429 — the per-minute burst bucket; Retry-After + details {limit}
|
|
46
56
|
"QUOTA_EXCEEDED", // 429 — a durable bound that WAITING clears; details {scope, limit, resetAt}
|
|
47
57
|
"INTERNAL_ERROR", // 500 — we broke it; the body is always generic, the cause stays server-side
|
|
@@ -66,6 +76,8 @@ export const ERROR_STATUS = {
|
|
|
66
76
|
UPSTREAM_BLOCKED: 422,
|
|
67
77
|
NAV_TIMEOUT: 422,
|
|
68
78
|
PROXY_REFUSED: 422,
|
|
79
|
+
FILE_TOO_LARGE: 422,
|
|
80
|
+
DOCUMENT_TIMEOUT: 422,
|
|
69
81
|
RATE_LIMIT_EXCEEDED: 429,
|
|
70
82
|
QUOTA_EXCEEDED: 429,
|
|
71
83
|
INTERNAL_ERROR: 500,
|
|
@@ -73,10 +85,11 @@ export const ERROR_STATUS = {
|
|
|
73
85
|
BILLING_UNAVAILABLE: 503,
|
|
74
86
|
};
|
|
75
87
|
/**
|
|
76
|
-
* The three no-charge outcomes, as DESIGN.md names them.
|
|
77
|
-
*
|
|
78
|
-
*
|
|
79
|
-
*
|
|
88
|
+
* The three no-charge outcomes, as DESIGN.md names them. Each code here maps to exactly one
|
|
89
|
+
* — this is the map the console reads to pick the hue and the word, and the SDK reads to
|
|
90
|
+
* tell a caller what to do next. A code absent from here gets no colour: the caller's own
|
|
91
|
+
* mistake (a bad body, a missing key), or a free refusal whose fix is in the request rather
|
|
92
|
+
* than the site (PROXY_REFUSED, FILE_TOO_LARGE, DOCUMENT_TIMEOUT).
|
|
80
93
|
*
|
|
81
94
|
* blocked → the site said no → try another engine, pool or country
|
|
82
95
|
* denied → our policy said no → change the URL; retrying cannot help
|
|
@@ -125,7 +138,8 @@ export const ENGINES = ["auto", "fetch", "browser"];
|
|
|
125
138
|
/** What a scrape can hand back. `markdown` is the default because it is what an LLM, a
|
|
126
139
|
* notebook and a human all read; `html` is the page as fetched; `text` is prose with the
|
|
127
140
|
* markup gone; `links` every absolute anchor in document order; `extract` is the page read
|
|
128
|
-
* into the fields the caller asked for, which is the one format a model writes
|
|
141
|
+
* into the fields the caller asked for, which is the one format a model writes; `raw` is the
|
|
142
|
+
* response's own bytes, the file a browser would save. */
|
|
129
143
|
export const SCRAPE_FORMATS = [
|
|
130
144
|
"markdown",
|
|
131
145
|
"html",
|
|
@@ -134,6 +148,7 @@ export const SCRAPE_FORMATS = [
|
|
|
134
148
|
"screenshot",
|
|
135
149
|
"pdf",
|
|
136
150
|
"extract",
|
|
151
|
+
"raw",
|
|
137
152
|
];
|
|
138
153
|
/** A session's life. `starting` is the driver bringing Chromium up; `live` is the only
|
|
139
154
|
* status whose `connectUrl` accepts a CDP client. The three end states answer one
|
|
@@ -146,7 +161,7 @@ export const SESSION_STATUSES = ["starting", "live", "closed", "expired", "faile
|
|
|
146
161
|
export const SEARCH_TOPICS = ["general", "news"];
|
|
147
162
|
/** Which front answered a search. Not a caller's choice — the ladder picks, and says which
|
|
148
163
|
* one it was, because a result set's flavour follows its index. */
|
|
149
|
-
export const SEARCH_ENGINES = ["bing", "brave"];
|
|
164
|
+
export const SEARCH_ENGINES = ["bing", "brave", "duckduckgo"];
|
|
150
165
|
/** A batch's life. `queued` is accepted but not yet picked up; `running` has at least one
|
|
151
166
|
* item in flight. The three end states answer the same question the session statuses do —
|
|
152
167
|
* who ended it: `completed` ran every item to a verdict (some of which may be failures,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"contract.js","sourceRoot":"","sources":["../../src/generated/contract.ts"],"names":[],"mappings":"AAAA,uDAAuD;AACvD,
|
|
1
|
+
{"version":3,"file":"contract.js","sourceRoot":"","sources":["../../src/generated/contract.ts"],"names":[],"mappings":"AAAA,uDAAuD;AACvD,gZAAgZ;AAChZ,EAAE;AACF,oEAAoE;AAEpE,wEAAwE;AAExE;;;;;;;;;;;GAWG;AACH,MAAM,CAAC,MAAM,WAAW,GAAG;IACvB,kBAAkB,EAAE,iFAAiF;IACrG,aAAa,EAAE,+DAA+D;IAC9E,aAAa,EAAE,sFAAsF;IACrG,cAAc,EAAE,gCAAgC;IAChD,eAAe,EAAE,6EAA6E;IAC9F,oFAAoF;IACpF,mFAAmF;IACnF,qFAAqF;IACrF,uFAAuF;IACvF,qFAAqF;IACrF,WAAW,EAAE,2EAA2E;IACxF,WAAW,EAAE,uCAAuC;IACpD,YAAY,EAAE,yEAAyE;IACvF,WAAW,EAAE,uEAAuE;IACpF,mBAAmB,EAAE,0DAA0D;IAC/E,UAAU,EAAE,uCAAuC;IACnD,mBAAmB,EAAE,qDAAqD;IAC1E,sFAAsF;IACtF,yFAAyF;IACzF,wFAAwF;IACxF,oFAAoF;IACpF,kBAAkB,EAAE,gEAAgE;IACpF,aAAa,EAAE,gEAAgE;IAC/E,wFAAwF;IACxF,uFAAuF;IACvF,yFAAyF;IACzF,eAAe,EAAE,iFAAiF;IAClG,wFAAwF;IACxF,uFAAuF;IACvF,yFAAyF;IACzF,gBAAgB,EAAE,kCAAkC;IACpD,yFAAyF;IACzF,wFAAwF;IACxF,sFAAsF;IACtF,iCAAiC;IACjC,kBAAkB,EAAE,8CAA8C;IAClE,qBAAqB,EAAE,mEAAmE;IAC1F,gBAAgB,EAAE,6EAA6E;IAC/F,gBAAgB,EAAE,6EAA6E;IAC/F,qBAAqB,EAAE,wFAAwF;IAC/G,qBAAqB,EAAE,uFAAuF;CACxG,CAAC;AAIX;2FAC2F;AAC3F,MAAM,CAAC,MAAM,YAAY,GAAG;IACxB,gBAAgB,EAAE,GAAG;IACrB,WAAW,EAAE,GAAG;IAChB,WAAW,EAAE,GAAG;IAChB,YAAY,EAAE,GAAG;IACjB,aAAa,EAAE,GAAG;IAClB,SAAS,EAAE,GAAG;IACd,SAAS,EAAE,GAAG;IACd,UAAU,EAAE,GAAG;IACf,SAAS,EAAE,GAAG;IACd,iBAAiB,EAAE,GAAG;IACtB,QAAQ,EAAE,GAAG;IACb,iBAAiB,EAAE,GAAG;IACtB,gBAAgB,EAAE,GAAG;IACrB,WAAW,EAAE,GAAG;IAChB,aAAa,EAAE,GAAG;IAClB,cAAc,EAAE,GAAG;IACnB,gBAAgB,EAAE,GAAG;IACrB,mBAAmB,EAAE,GAAG;IACxB,cAAc,EAAE,GAAG;IACnB,cAAc,EAAE,GAAG;IACnB,mBAAmB,EAAE,GAAG;IACxB,mBAAmB,EAAE,GAAG;CACkB,CAAC;AAE/C;;;;;;;;;;GAUG;AACH,MAAM,CAAC,MAAM,UAAU,GAAgE;IACnF,gBAAgB,EAAE,SAAS;IAC3B,WAAW,EAAE,SAAS;IACtB,WAAW,EAAE,QAAQ;IACrB,UAAU,EAAE,QAAQ;IACpB,cAAc,EAAE,QAAQ;IACxB,mBAAmB,EAAE,QAAQ;IAC7B,qFAAqF;IACrF,uFAAuF;IACvF,mBAAmB,EAAE,QAAQ;CAChC,CAAC;AAEF;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,OAAO,GAAG;IACnB,uEAAuE;IACvE,IAAI,EAAE,gBAAgB;IACtB,+EAA+E;IAC/E,gBAAgB,EAAE,qBAAqB;IACvC,SAAS,EAAE,cAAc;IACzB,sEAAsE;IACtE,MAAM,EAAE,iBAAiB;IACzB,yCAAyC;IACzC,MAAM,EAAE,iBAAiB;IACzB,UAAU,EAAE,aAAa;CACnB,CAAC;AAuCX,wEAAwE;AAExE;;;wDAGwD;AACxD,MAAM,CAAC,MAAM,WAAW,GAAG,CAAC,aAAa,EAAE,QAAQ,CAAU,CAAC;AAI9D;iFACiF;AACjF,MAAM,CAAC,MAAM,OAAO,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,SAAS,CAAU,CAAC;AAO7D;;;;2DAI2D;AAC3D,MAAM,CAAC,MAAM,cAAc,GAAG;IAC1B,UAAU;IACV,MAAM;IACN,MAAM;IACN,OAAO;IACP,YAAY;IACZ,KAAK;IACL,SAAS;IACT,KAAK;CACC,CAAC;AAIX;;;;4EAI4E;AAC5E,MAAM,CAAC,MAAM,gBAAgB,GAAG,CAAC,UAAU,EAAE,MAAM,EAAE,QAAQ,EAAE,SAAS,EAAE,QAAQ,CAAU,CAAC;AAI7F;qFACqF;AACrF,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,SAAS,EAAE,MAAM,CAAU,CAAC;AAI1D;oEACoE;AACpE,MAAM,CAAC,MAAM,cAAc,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,YAAY,CAAU,CAAC;AAOvE;;;;mEAImE;AACnE,MAAM,CAAC,MAAM,cAAc,GAAG,CAAC,QAAQ,EAAE,SAAS,EAAE,WAAW,EAAE,UAAU,EAAE,QAAQ,CAAU,CAAC;AAIhG;;;;;eAKe;AACf,MAAM,CAAC,MAAM,mBAAmB,GAAG,CAAC,QAAQ,EAAE,SAAS,EAAE,MAAM,EAAE,QAAQ,EAAE,SAAS,CAAU,CAAC;AAI/F;;;;;;8BAM8B;AAC9B,MAAM,CAAC,MAAM,cAAc,GAAG,CAAC,iBAAiB,EAAE,sBAAsB,CAAU,CAAC;AAInF;;;kBAGkB;AAClB,MAAM,CAAC,MAAM,yBAAyB,GAAG,CAAC,SAAS,EAAE,WAAW,EAAE,QAAQ,CAAU,CAAC"}
|