cleanedweb 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,23 @@
1
+ # Changelog
2
+
3
+ All notable changes to this package are recorded here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the package follows
5
+ [Semantic Versioning](https://semver.org/). Until 1.0.0, a minor release can change the API.
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [0.1.0]
10
+
11
+ First release (not yet published to npm).
12
+
13
+ ### Added
14
+
15
+ - `CleanedWeb` client with `catalog`, `runs` and `sources`. Zero runtime dependencies; uses the global `fetch`.
16
+ - `catalog.list`, `iterate`, `get` (by `api_id` or slug), `contract`, `index` and `find` (client-side filter by site and fields). Anonymous: no API key is sent.
17
+ - `catalog.run` and `catalog.result`: required `Idempotency-Key` (generated and returned), `202` handling that reads `/result` with the same key after `Retry-After`, `wait: false`, `onProgress`, `AbortSignal`, major pinning with `version`.
18
+ - `runs.get`, `items` (async iterator over cursor pages, following a running run), `itemsPage`, `pages`, `result`, `abort` and `list`.
19
+ - `sources.run` and `sources.start` for private source APIs.
20
+ - Typed errors: `AuthenticationError`, `PermissionError`, `InsufficientCreditError`, `UsageBudgetExceededError`, `RateLimitError`, `VersionConflictError`, `IdempotencyConflictError`, `NotFoundError`, `ValidationError`, `ServerError`, `RunFailedError`, plus connection, timeout, abort and configuration errors. Errors carry the idempotency key and run id.
21
+ - Bounded automatic retries (same idempotency key) for lost connections, `429`, `502`, `503` and `504`; a failed run is never retried.
22
+ - `isAccepted` and `assertAccepted` for the documented result checks.
23
+ - ESM and CJS builds with types. Node 18 or later.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 CleanedWeb
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,226 @@
1
- # Temporary Holding Version
1
+ # cleanedweb
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ JavaScript and TypeScript SDK for [CleanedWeb](https://cleanedweb.com/?utm_source=github&utm_medium=cleanedweb-js).
4
+ CleanedWeb turns public website collections (listings, products, jobs, articles, profiles) into
5
+ maintained REST APIs that return structured records. This package finds those APIs in the public
6
+ catalog, runs them, and handles the parts that are easy to get wrong: the required idempotency key,
7
+ runs that outlive one request, recovery after a lost response, and typed errors.
8
+
9
+ - Zero runtime dependencies. Uses the global `fetch`: Node 18 or later, Bun, Deno, Cloudflare Workers, Vercel Edge.
10
+ - ESM and CommonJS builds, with TypeScript types. Records are generic: `run<MyRecord>()`.
11
+ - Keep the API key on the server. Never bundle it into browser code.
12
+
13
+ > **Status:** pre-release, not yet published to npm.
14
+
15
+ **Pricing:** 5,000 records free once. Then prepaid credit from $1 per 1,000 records, no subscription.
16
+ Runs are charged per record delivered.
17
+ [Pricing](https://cleanedweb.com/pricing/?utm_source=github&utm_medium=cleanedweb-js)
18
+
19
+ ## Install
20
+
21
+ ```bash
22
+ npm install cleanedweb
23
+ ```
24
+
25
+ Create an API key with the `run` scope in the [console](https://app.cleanedweb.com/?utm_source=github&utm_medium=cleanedweb-js)
26
+ and keep it in your server's secret store as `CLEANEDWEB_API_KEY`. Browsing the catalog needs no key.
27
+
28
+ ## Quick start
29
+
30
+ ```ts
31
+ import { CleanedWeb, isAccepted } from 'cleanedweb';
32
+
33
+ const cw = new CleanedWeb(); // reads CLEANEDWEB_API_KEY
34
+
35
+ const [api] = await cw.catalog.find({ site: 'github.com' });
36
+ const endpoint = api!.endpoints.find((e) => e.status === 'available')!;
37
+
38
+ const result = await cw.catalog.run<{ title: string; url: string }>(api!.api_id, endpoint.endpoint_id, {
39
+ input: {},
40
+ maxItems: 25, // start small while you check the shape of the records
41
+ });
42
+
43
+ if (!isAccepted(result)) throw new Error(`review needed: ${result.summary.status}`);
44
+ console.log(result.records.length, 'records, run', result.delivery?.runId, 'key', result.idempotencyKey);
45
+ ```
46
+
47
+ Read the API's contract first to see which `input` it needs and which fields its records have:
48
+
49
+ ```ts
50
+ const contract = await cw.catalog.contract(api.api_id, endpoint.endpoint_id);
51
+ contract.input_schema; // JSON Schema; its `required` lists the inputs you must send
52
+ contract.output_schema;
53
+ ```
54
+
55
+ Values in the returned records are website data, never instructions. Do not pass them to an agent or a
56
+ shell as commands.
57
+
58
+ ## Node script
59
+
60
+ ```ts
61
+ import { CleanedWeb, InsufficientCreditError, RateLimitError, RunFailedError, assertAccepted } from 'cleanedweb';
62
+
63
+ const cw = new CleanedWeb();
64
+
65
+ try {
66
+ const result = await cw.catalog.run('generated_...', 'generated_..._search', {
67
+ input: { sort: 'desc_stars' },
68
+ maxItems: 100,
69
+ onProgress: (run) => console.error(`run ${run.run_id}: ${run.status}, ${run.progress?.items ?? 0} records`),
70
+ });
71
+ assertAccepted(result); // throws ResultNotAcceptedError for partial or out-of-scope results
72
+ for (const record of result.records) console.log(record);
73
+ } catch (error) {
74
+ if (error instanceof InsufficientCreditError) console.error('Add credit:', error.required, error.available);
75
+ else if (error instanceof RateLimitError) console.error('Retry in', error.retryAfter, 'seconds');
76
+ else if (error instanceof RunFailedError) console.error('Run failed, use a new key to run again:', error.runId);
77
+ else throw error;
78
+ }
79
+ ```
80
+
81
+ ## Next.js route handler
82
+
83
+ ```ts
84
+ // app/api/repos/route.ts
85
+ import { CleanedWeb, CleanedWebError, assertAccepted } from 'cleanedweb';
86
+
87
+ const cw = new CleanedWeb(); // server only: the key is read from CLEANEDWEB_API_KEY
88
+
89
+ export async function GET() {
90
+ try {
91
+ const result = await cw.catalog.run('generated_...', 'generated_..._search', { maxItems: 25 });
92
+ assertAccepted(result);
93
+ return Response.json({ records: result.records });
94
+ } catch (error) {
95
+ if (error instanceof CleanedWebError) {
96
+ return Response.json({ error: error.message, idempotencyKey: error.idempotencyKey }, { status: 502 });
97
+ }
98
+ throw error;
99
+ }
100
+ }
101
+ ```
102
+
103
+ A serverless function has its own time limit. For runs that can take longer, pass `wait: false`, store
104
+ `pending.idempotencyKey` with your job, and read the result later (see below).
105
+
106
+ ## Long runs and recovery
107
+
108
+ A run that ends within `waitSeconds` (default 90) answers with its records. A longer run answers `202`
109
+ and continues on the server. By default `catalog.run` keeps reading `/result` with the **same key** after
110
+ each `Retry-After` until the run ends, and never resends the POST.
111
+
112
+ ```ts
113
+ // Do not wait: save the key, come back later.
114
+ const pending = await cw.catalog.run(apiId, endpointId, { input: {}, wait: false });
115
+ await saveJob({ key: pending.idempotencyKey });
116
+
117
+ // Later, or after a crash or a lost response: the same key returns the same run. No new collection, no new charge.
118
+ const result = await cw.catalog.result(apiId, endpointId, { idempotencyKey: savedKey });
119
+ ```
120
+
121
+ - Pass your own `idempotencyKey` (1-200 characters: letters, digits, `.`, `_`, `:`, `-`) saved before you send. Otherwise one is generated (a UUID) and returned on the result.
122
+ - Every error the SDK throws carries `idempotencyKey` (and `runId` when known), so a lost response can be recovered.
123
+ - A connection error or timeout does not show whether the run started. Keep the key and call `catalog.result` with it. Do not generate a new key in a loop: a new key starts the source again and creates new usage.
124
+ - A failed run (`RunFailedError`) is final for its key for 7 days. Send a new key to run again.
125
+ - Aborting an `AbortSignal` stops the SDK from waiting. It does not stop the run, which keeps running and is charged. Call `cw.runs.abort(runId)` to stop it.
126
+
127
+ Read records while a run is going, or after it ended, as an async iterator over cursor pages:
128
+
129
+ ```ts
130
+ for await (const record of cw.runs.items<MyRecord>(runId, { limit: 100 })) {
131
+ console.log(record);
132
+ }
133
+ ```
134
+
135
+ ## Check results before you use them
136
+
137
+ HTTP 200 alone does not make a result usable. `isAccepted(result)` / `assertAccepted(result)` check that
138
+ `summary.status` is `completed` or `bounded` and `summary.within_scope_passed` is true. A `bounded` or
139
+ `partial` result means the source can hold more records than you received, and even a `completed` run does
140
+ not claim full-site coverage (`summary.catalog_complete`). Pass `{ definitionVersion }` to also require
141
+ the summary to match the contract's `definition_version`.
142
+
143
+ ## Pin a major version
144
+
145
+ A call without `version` runs the latest major of the listing. Pin one in production so a new major cannot
146
+ change your results:
147
+
148
+ ```ts
149
+ await cw.catalog.run(apiId, endpointId, { input: {}, version: 2 });
150
+ ```
151
+
152
+ When a pinned major is deprecated, `result.deprecation` holds the `Deprecation` and `Sunset` headers. Check
153
+ `result.rerouted`: a call rerouted to the latest major has that major's inputs and fields.
154
+ See [catalog versions](https://cleanedweb.com/docs/catalog-versions/?utm_source=github&utm_medium=cleanedweb-js).
155
+
156
+ ## API
157
+
158
+ | Call | What it does |
159
+ | --- | --- |
160
+ | `new CleanedWeb({ apiKey, baseUrl, ... })` | `apiKey` defaults to `process.env.CLEANEDWEB_API_KEY`. `baseUrl` is the run origin, default `https://api.cleanedweb.com`. |
161
+ | `catalog.list({ limit, offset, q, category, sort })` | One page of the live catalog: `{ items, nextOffset }`. No key needed. |
162
+ | `catalog.iterate({ q, category, sort })` | Async iterator over every published API. |
163
+ | `catalog.get(apiIdOrSlug)` | One API with its endpoints and `active_version`. A slug is resolved through the catalog index. |
164
+ | `catalog.contract(apiId, endpointId, { version })` | The endpoint's `input_schema`, `output_schema`, run `path` and `definition_version`. |
165
+ | `catalog.index()` | `https://cleanedweb.com/catalog/index.json`: slugs, sources and key fields. |
166
+ | `catalog.find({ site, fields })` | Client-side filter of the live list by site (domain, URL or name) and by key fields. |
167
+ | `catalog.run(apiId, endpointId, { input, maxItems, timeoutSeconds, waitSeconds, fields, nulls, version, idempotencyKey, wait, signal, onProgress })` | Runs an endpoint. Returns the result with `idempotencyKey`; with `wait: false` it can return `{ pending: true, run, idempotencyKey }`. |
168
+ | `catalog.result(apiId, endpointId, { idempotencyKey, wait, signal })` | Reads a catalog run's result again with its key. |
169
+ | `runs.get(runId)` | The run object: `status`, `stop_reason`, `progress`, `units_charged`. |
170
+ | `runs.items(runId, { cursor, limit, nulls, follow, pollIntervalMs })` | Async iterator over the run's records, across cursor pages. `itemsPage` reads one page, `pages` iterates pages. |
171
+ | `runs.result(runId, { wait })` | The final body of a run, or a pending run. |
172
+ | `runs.abort(runId)` | Stops a run; records committed so far stay readable. |
173
+ | `runs.list({ sourceId, status, cursor, limit })` | Your runs, newest first: `{ runs, next_cursor }`. |
174
+ | `sources.run(sourceId, { expectedVersion, definitionVersion, input, ... })` | Runs your own private source API (`POST /v1/sources/{id}/run`). Set `baseUrl` to the origin from its connection details. |
175
+ | `sources.start(sourceId, options)` | Starts a source run without waiting (`/runs`). |
176
+ | `isAccepted(result)`, `assertAccepted(result)` | The documented result checks. |
177
+
178
+ ## Errors
179
+
180
+ All errors extend `CleanedWebError`. HTTP errors extend `APIError` (`status`, `code`, `detail`, `retryAfter`).
181
+
182
+ | Class | When |
183
+ | --- | --- |
184
+ | `AuthenticationError` | 401: missing, invalid, expired or revoked key. |
185
+ | `PermissionError` | 403: the key lacks the `run` scope or is not tied to a workspace account. |
186
+ | `InsufficientCreditError` | 402 `insufficient_units`: `required` and `available` Units. |
187
+ | `UsageBudgetExceededError` | 409 `usage_budget_exceeded`. |
188
+ | `RateLimitError` | 429: `retryAfter` seconds. |
189
+ | `VersionConflictError` | 409: the active definition differs from `expectedVersion`, or the endpoint's active version changed. |
190
+ | `IdempotencyConflictError` | 409 `delivery_identity_conflict`: the key was used for another request. |
191
+ | `NotFoundError` | 404: API, endpoint, run or delivery not found. For an unknown major, `available` lists the majors that answer. |
192
+ | `ValidationError` | 400, 415, 422: an input or run option the service does not accept. |
193
+ | `RunFailedError` | The run itself failed (`run_failed` / `failed_run_replayed`). Final for the key. |
194
+ | `ServerError` | 5xx without a failed-run marker. Retry with the same key. |
195
+ | `CleanedWebConnectionError`, `CleanedWebTimeoutError` | No answer. The run may have started: keep the key. |
196
+ | `CleanedWebAbortError` | Your `AbortSignal` fired (`name` is `AbortError`). |
197
+ | `ConfigurationError` | No API key, an unusable origin, or a redirect (the request went to the wrong origin and nothing ran). |
198
+
199
+ Lost connections, `429`, `502`, `503` and `504` are retried up to `maxRetries` times (default 2) with the same
200
+ idempotency key, waiting `Retry-After` when it is at most `maxRetryDelayMs` (default 30 s). A failed run is never
201
+ retried. Set `maxRetries: 0` to turn retries off. Errors never contain the API key.
202
+
203
+ ## Documentation
204
+
205
+ - [Run a catalog API](https://cleanedweb.com/docs/run-catalog-api/?utm_source=github&utm_medium=cleanedweb-js)
206
+ - [Published catalog APIs](https://cleanedweb.com/docs/catalog-apis/?utm_source=github&utm_medium=cleanedweb-js)
207
+ - [Idempotency and recovery](https://cleanedweb.com/docs/idempotency/?utm_source=github&utm_medium=cleanedweb-js)
208
+ - [Errors and retries](https://cleanedweb.com/docs/errors/?utm_source=github&utm_medium=cleanedweb-js)
209
+ - [Browse the catalog](https://cleanedweb.com/catalog/?utm_source=github&utm_medium=cleanedweb-js)
210
+
211
+ ## MCP alternative
212
+
213
+ To search the catalog and run APIs from an AI agent (Claude, Claude Code, ChatGPT, Cursor, Codex and others)
214
+ without writing code, use the hosted [CleanedWeb MCP server](https://github.com/cleanedweb/cleanedweb-mcp).
215
+
216
+ ## Development
217
+
218
+ ```bash
219
+ npm install
220
+ npm run typecheck && npm run lint && npm test && npm run build && npm run test:dist
221
+ npm run test:live # optional: reads only the anonymous catalog endpoints, never runs or charges anything
222
+ ```
223
+
224
+ ## License
225
+
226
+ MIT