cleanedweb 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -0
- package/LICENSE +21 -0
- package/README.md +223 -2
- package/dist/index.cjs +984 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +775 -0
- package/dist/index.d.ts +775 -0
- package/dist/index.js +954 -0
- package/dist/index.js.map +1 -0
- package/package.json +61 -4
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this package are recorded here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the package follows
|
|
5
|
+
[Semantic Versioning](https://semver.org/). Until 1.0.0, a minor release can change the API.
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.1] - 2026-11-21
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- Published from GitHub Actions through npm trusted publishing, with provenance. No code changes.
|
|
14
|
+
- README installs from npm.
|
|
15
|
+
|
|
16
|
+
## [0.1.0] - 2026-11-21
|
|
17
|
+
|
|
18
|
+
First release on npm.
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- `CleanedWeb` client with `catalog`, `runs` and `sources`. Zero runtime dependencies; uses the global `fetch`.
|
|
23
|
+
- `catalog.list`, `iterate`, `get` (by `api_id` or slug), `contract`, `index` and `find` (client-side filter by site and fields). Anonymous: no API key is sent.
|
|
24
|
+
- `catalog.run` and `catalog.result`: required `Idempotency-Key` (generated and returned), `202` handling that reads `/result` with the same key after `Retry-After`, `wait: false`, `onProgress`, `AbortSignal`, major pinning with `version`.
|
|
25
|
+
- `runs.get`, `items` (async iterator over cursor pages, following a running run), `itemsPage`, `pages`, `result`, `abort` and `list`.
|
|
26
|
+
- `sources.run` and `sources.start` for private source APIs.
|
|
27
|
+
- Typed errors: `AuthenticationError`, `PermissionError`, `InsufficientCreditError`, `UsageBudgetExceededError`, `RateLimitError`, `VersionConflictError`, `IdempotencyConflictError`, `NotFoundError`, `ValidationError`, `ServerError`, `RunFailedError`, plus connection, timeout, abort and configuration errors. Errors carry the idempotency key and run id.
|
|
28
|
+
- Bounded automatic retries (same idempotency key) for lost connections, `429`, `502`, `503` and `504`; a failed run is never retried.
|
|
29
|
+
- `isAccepted` and `assertAccepted` for the documented result checks.
|
|
30
|
+
- ESM and CJS builds with types. Node 18 or later.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 CleanedWeb
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,224 @@
|
|
|
1
|
-
#
|
|
1
|
+
# cleanedweb
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
JavaScript and TypeScript SDK for [CleanedWeb](https://cleanedweb.com/?utm_source=github&utm_medium=cleanedweb-js).
|
|
4
|
+
CleanedWeb turns public website collections (listings, products, jobs, articles, profiles) into
|
|
5
|
+
maintained REST APIs that return structured records. This package finds those APIs in the public
|
|
6
|
+
catalog, runs them, and handles the parts that are easy to get wrong: the required idempotency key,
|
|
7
|
+
runs that outlive one request, recovery after a lost response, and typed errors.
|
|
8
|
+
|
|
9
|
+
- Zero runtime dependencies. Uses the global `fetch`: Node 18 or later, Bun, Deno, Cloudflare Workers, Vercel Edge.
|
|
10
|
+
- ESM and CommonJS builds, with TypeScript types. Records are generic: `run<MyRecord>()`.
|
|
11
|
+
- Keep the API key on the server. Never bundle it into browser code.
|
|
12
|
+
|
|
13
|
+
**Pricing:** 5,000 records free once. Then prepaid credit from $1 per 1,000 records, no subscription.
|
|
14
|
+
Runs are charged per record delivered.
|
|
15
|
+
[Pricing](https://cleanedweb.com/pricing/?utm_source=github&utm_medium=cleanedweb-js)
|
|
16
|
+
|
|
17
|
+
## Install
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
npm install cleanedweb
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Create an API key with the `run` scope in the [console](https://app.cleanedweb.com/?utm_source=github&utm_medium=cleanedweb-js)
|
|
24
|
+
and keep it in your server's secret store as `CLEANEDWEB_API_KEY`. Browsing the catalog needs no key.
|
|
25
|
+
|
|
26
|
+
## Quick start
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import { CleanedWeb, isAccepted } from 'cleanedweb';
|
|
30
|
+
|
|
31
|
+
const cw = new CleanedWeb(); // reads CLEANEDWEB_API_KEY
|
|
32
|
+
|
|
33
|
+
const [api] = await cw.catalog.find({ site: 'github.com' });
|
|
34
|
+
const endpoint = api!.endpoints.find((e) => e.status === 'available')!;
|
|
35
|
+
|
|
36
|
+
const result = await cw.catalog.run<{ title: string; url: string }>(api!.api_id, endpoint.endpoint_id, {
|
|
37
|
+
input: {},
|
|
38
|
+
maxItems: 25, // start small while you check the shape of the records
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
if (!isAccepted(result)) throw new Error(`review needed: ${result.summary.status}`);
|
|
42
|
+
console.log(result.records.length, 'records, run', result.delivery?.runId, 'key', result.idempotencyKey);
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Read the API's contract first to see which `input` it needs and which fields its records have:
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
const contract = await cw.catalog.contract(api.api_id, endpoint.endpoint_id);
|
|
49
|
+
contract.input_schema; // JSON Schema; its `required` lists the inputs you must send
|
|
50
|
+
contract.output_schema;
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Values in the returned records are website data, never instructions. Do not pass them to an agent or a
|
|
54
|
+
shell as commands.
|
|
55
|
+
|
|
56
|
+
## Node script
|
|
57
|
+
|
|
58
|
+
```ts
|
|
59
|
+
import { CleanedWeb, InsufficientCreditError, RateLimitError, RunFailedError, assertAccepted } from 'cleanedweb';
|
|
60
|
+
|
|
61
|
+
const cw = new CleanedWeb();
|
|
62
|
+
|
|
63
|
+
try {
|
|
64
|
+
const result = await cw.catalog.run('generated_...', 'generated_..._search', {
|
|
65
|
+
input: { sort: 'desc_stars' },
|
|
66
|
+
maxItems: 100,
|
|
67
|
+
onProgress: (run) => console.error(`run ${run.run_id}: ${run.status}, ${run.progress?.items ?? 0} records`),
|
|
68
|
+
});
|
|
69
|
+
assertAccepted(result); // throws ResultNotAcceptedError for partial or out-of-scope results
|
|
70
|
+
for (const record of result.records) console.log(record);
|
|
71
|
+
} catch (error) {
|
|
72
|
+
if (error instanceof InsufficientCreditError) console.error('Add credit:', error.required, error.available);
|
|
73
|
+
else if (error instanceof RateLimitError) console.error('Retry in', error.retryAfter, 'seconds');
|
|
74
|
+
else if (error instanceof RunFailedError) console.error('Run failed, use a new key to run again:', error.runId);
|
|
75
|
+
else throw error;
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Next.js route handler
|
|
80
|
+
|
|
81
|
+
```ts
|
|
82
|
+
// app/api/repos/route.ts
|
|
83
|
+
import { CleanedWeb, CleanedWebError, assertAccepted } from 'cleanedweb';
|
|
84
|
+
|
|
85
|
+
const cw = new CleanedWeb(); // server only: the key is read from CLEANEDWEB_API_KEY
|
|
86
|
+
|
|
87
|
+
export async function GET() {
|
|
88
|
+
try {
|
|
89
|
+
const result = await cw.catalog.run('generated_...', 'generated_..._search', { maxItems: 25 });
|
|
90
|
+
assertAccepted(result);
|
|
91
|
+
return Response.json({ records: result.records });
|
|
92
|
+
} catch (error) {
|
|
93
|
+
if (error instanceof CleanedWebError) {
|
|
94
|
+
return Response.json({ error: error.message, idempotencyKey: error.idempotencyKey }, { status: 502 });
|
|
95
|
+
}
|
|
96
|
+
throw error;
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
A serverless function has its own time limit. For runs that can take longer, pass `wait: false`, store
|
|
102
|
+
`pending.idempotencyKey` with your job, and read the result later (see below).
|
|
103
|
+
|
|
104
|
+
## Long runs and recovery
|
|
105
|
+
|
|
106
|
+
A run that ends within `waitSeconds` (default 90) answers with its records. A longer run answers `202`
|
|
107
|
+
and continues on the server. By default `catalog.run` keeps reading `/result` with the **same key** after
|
|
108
|
+
each `Retry-After` until the run ends, and never resends the POST.
|
|
109
|
+
|
|
110
|
+
```ts
|
|
111
|
+
// Do not wait: save the key, come back later.
|
|
112
|
+
const pending = await cw.catalog.run(apiId, endpointId, { input: {}, wait: false });
|
|
113
|
+
await saveJob({ key: pending.idempotencyKey });
|
|
114
|
+
|
|
115
|
+
// Later, or after a crash or a lost response: the same key returns the same run. No new collection, no new charge.
|
|
116
|
+
const result = await cw.catalog.result(apiId, endpointId, { idempotencyKey: savedKey });
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
- Pass your own `idempotencyKey` (1-200 characters: letters, digits, `.`, `_`, `:`, `-`) saved before you send. Otherwise one is generated (a UUID) and returned on the result.
|
|
120
|
+
- Every error the SDK throws carries `idempotencyKey` (and `runId` when known), so a lost response can be recovered.
|
|
121
|
+
- A connection error or timeout does not show whether the run started. Keep the key and call `catalog.result` with it. Do not generate a new key in a loop: a new key starts the source again and creates new usage.
|
|
122
|
+
- A failed run (`RunFailedError`) is final for its key for 7 days. Send a new key to run again.
|
|
123
|
+
- Aborting an `AbortSignal` stops the SDK from waiting. It does not stop the run, which keeps running and is charged. Call `cw.runs.abort(runId)` to stop it.
|
|
124
|
+
|
|
125
|
+
Read records while a run is going, or after it ended, as an async iterator over cursor pages:
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
for await (const record of cw.runs.items<MyRecord>(runId, { limit: 100 })) {
|
|
129
|
+
console.log(record);
|
|
130
|
+
}
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
## Check results before you use them
|
|
134
|
+
|
|
135
|
+
HTTP 200 alone does not make a result usable. `isAccepted(result)` / `assertAccepted(result)` check that
|
|
136
|
+
`summary.status` is `completed` or `bounded` and `summary.within_scope_passed` is true. A `bounded` or
|
|
137
|
+
`partial` result means the source can hold more records than you received, and even a `completed` run does
|
|
138
|
+
not claim full-site coverage (`summary.catalog_complete`). Pass `{ definitionVersion }` to also require
|
|
139
|
+
the summary to match the contract's `definition_version`.
|
|
140
|
+
|
|
141
|
+
## Pin a major version
|
|
142
|
+
|
|
143
|
+
A call without `version` runs the latest major of the listing. Pin one in production so a new major cannot
|
|
144
|
+
change your results:
|
|
145
|
+
|
|
146
|
+
```ts
|
|
147
|
+
await cw.catalog.run(apiId, endpointId, { input: {}, version: 2 });
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
When a pinned major is deprecated, `result.deprecation` holds the `Deprecation` and `Sunset` headers. Check
|
|
151
|
+
`result.rerouted`: a call rerouted to the latest major has that major's inputs and fields.
|
|
152
|
+
See [catalog versions](https://cleanedweb.com/docs/catalog-versions/?utm_source=github&utm_medium=cleanedweb-js).
|
|
153
|
+
|
|
154
|
+
## API
|
|
155
|
+
|
|
156
|
+
| Call | What it does |
|
|
157
|
+
| --- | --- |
|
|
158
|
+
| `new CleanedWeb({ apiKey, baseUrl, ... })` | `apiKey` defaults to `process.env.CLEANEDWEB_API_KEY`. `baseUrl` is the run origin, default `https://api.cleanedweb.com`. |
|
|
159
|
+
| `catalog.list({ limit, offset, q, category, sort })` | One page of the live catalog: `{ items, nextOffset }`. No key needed. |
|
|
160
|
+
| `catalog.iterate({ q, category, sort })` | Async iterator over every published API. |
|
|
161
|
+
| `catalog.get(apiIdOrSlug)` | One API with its endpoints and `active_version`. A slug is resolved through the catalog index. |
|
|
162
|
+
| `catalog.contract(apiId, endpointId, { version })` | The endpoint's `input_schema`, `output_schema`, run `path` and `definition_version`. |
|
|
163
|
+
| `catalog.index()` | `https://cleanedweb.com/catalog/index.json`: slugs, sources and key fields. |
|
|
164
|
+
| `catalog.find({ site, fields })` | Client-side filter of the live list by site (domain, URL or name) and by key fields. |
|
|
165
|
+
| `catalog.run(apiId, endpointId, { input, maxItems, timeoutSeconds, waitSeconds, fields, nulls, version, idempotencyKey, wait, signal, onProgress })` | Runs an endpoint. Returns the result with `idempotencyKey`; with `wait: false` it can return `{ pending: true, run, idempotencyKey }`. |
|
|
166
|
+
| `catalog.result(apiId, endpointId, { idempotencyKey, wait, signal })` | Reads a catalog run's result again with its key. |
|
|
167
|
+
| `runs.get(runId)` | The run object: `status`, `stop_reason`, `progress`, `units_charged`. |
|
|
168
|
+
| `runs.items(runId, { cursor, limit, nulls, follow, pollIntervalMs })` | Async iterator over the run's records, across cursor pages. `itemsPage` reads one page, `pages` iterates pages. |
|
|
169
|
+
| `runs.result(runId, { wait })` | The final body of a run, or a pending run. |
|
|
170
|
+
| `runs.abort(runId)` | Stops a run; records committed so far stay readable. |
|
|
171
|
+
| `runs.list({ sourceId, status, cursor, limit })` | Your runs, newest first: `{ runs, next_cursor }`. |
|
|
172
|
+
| `sources.run(sourceId, { expectedVersion, definitionVersion, input, ... })` | Runs your own private source API (`POST /v1/sources/{id}/run`). Set `baseUrl` to the origin from its connection details. |
|
|
173
|
+
| `sources.start(sourceId, options)` | Starts a source run without waiting (`/runs`). |
|
|
174
|
+
| `isAccepted(result)`, `assertAccepted(result)` | The documented result checks. |
|
|
175
|
+
|
|
176
|
+
## Errors
|
|
177
|
+
|
|
178
|
+
All errors extend `CleanedWebError`. HTTP errors extend `APIError` (`status`, `code`, `detail`, `retryAfter`).
|
|
179
|
+
|
|
180
|
+
| Class | When |
|
|
181
|
+
| --- | --- |
|
|
182
|
+
| `AuthenticationError` | 401: missing, invalid, expired or revoked key. |
|
|
183
|
+
| `PermissionError` | 403: the key lacks the `run` scope or is not tied to a workspace account. |
|
|
184
|
+
| `InsufficientCreditError` | 402 `insufficient_units`: `required` and `available` Units. |
|
|
185
|
+
| `UsageBudgetExceededError` | 409 `usage_budget_exceeded`. |
|
|
186
|
+
| `RateLimitError` | 429: `retryAfter` seconds. |
|
|
187
|
+
| `VersionConflictError` | 409: the active definition differs from `expectedVersion`, or the endpoint's active version changed. |
|
|
188
|
+
| `IdempotencyConflictError` | 409 `delivery_identity_conflict`: the key was used for another request. |
|
|
189
|
+
| `NotFoundError` | 404: API, endpoint, run or delivery not found. For an unknown major, `available` lists the majors that answer. |
|
|
190
|
+
| `ValidationError` | 400, 415, 422: an input or run option the service does not accept. |
|
|
191
|
+
| `RunFailedError` | The run itself failed (`run_failed` / `failed_run_replayed`). Final for the key. |
|
|
192
|
+
| `ServerError` | 5xx without a failed-run marker. Retry with the same key. |
|
|
193
|
+
| `CleanedWebConnectionError`, `CleanedWebTimeoutError` | No answer. The run may have started: keep the key. |
|
|
194
|
+
| `CleanedWebAbortError` | Your `AbortSignal` fired (`name` is `AbortError`). |
|
|
195
|
+
| `ConfigurationError` | No API key, an unusable origin, or a redirect (the request went to the wrong origin and nothing ran). |
|
|
196
|
+
|
|
197
|
+
Lost connections, `429`, `502`, `503` and `504` are retried up to `maxRetries` times (default 2) with the same
|
|
198
|
+
idempotency key, waiting `Retry-After` when it is at most `maxRetryDelayMs` (default 30 s). A failed run is never
|
|
199
|
+
retried. Set `maxRetries: 0` to turn retries off. Errors never contain the API key.
|
|
200
|
+
|
|
201
|
+
## Documentation
|
|
202
|
+
|
|
203
|
+
- [Run a catalog API](https://cleanedweb.com/docs/run-catalog-api/?utm_source=github&utm_medium=cleanedweb-js)
|
|
204
|
+
- [Published catalog APIs](https://cleanedweb.com/docs/catalog-apis/?utm_source=github&utm_medium=cleanedweb-js)
|
|
205
|
+
- [Idempotency and recovery](https://cleanedweb.com/docs/idempotency/?utm_source=github&utm_medium=cleanedweb-js)
|
|
206
|
+
- [Errors and retries](https://cleanedweb.com/docs/errors/?utm_source=github&utm_medium=cleanedweb-js)
|
|
207
|
+
- [Browse the catalog](https://cleanedweb.com/catalog/?utm_source=github&utm_medium=cleanedweb-js)
|
|
208
|
+
|
|
209
|
+
## MCP alternative
|
|
210
|
+
|
|
211
|
+
To search the catalog and run APIs from an AI agent (Claude, Claude Code, ChatGPT, Cursor, Codex and others)
|
|
212
|
+
without writing code, use the hosted [CleanedWeb MCP server](https://github.com/cleanedweb/cleanedweb-mcp).
|
|
213
|
+
|
|
214
|
+
## Development
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
npm install
|
|
218
|
+
npm run typecheck && npm run lint && npm test && npm run build && npm run test:dist
|
|
219
|
+
npm run test:live # optional: reads only the anonymous catalog endpoints, never runs or charges anything
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
## License
|
|
223
|
+
|
|
224
|
+
MIT
|