awwwards-mcp 1.6.0 → 1.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.dev.md ADDED
@@ -0,0 +1,177 @@
1
+ # awwwards-mcp — project & architecture guide
2
+
3
+ Developer-facing guide to this repository. The [main README](README.md) covers
4
+ installation and user-facing features; this file explains how the codebase is
5
+ structured, how the data flows, and how to work on it without breaking it.
6
+
7
+ ## What this is
8
+
9
+ A free, open-source MCP server that gives AI agents design inspiration from
10
+ [Awwwards](https://www.awwwards.com/). No API key, no account: it politely
11
+ scrapes public pages (1 request/second, robots.txt-compliant paths), caches
12
+ them in local SQLite, and serves them through 7 MCP tools — screenshots,
13
+ design DNA, band maps and motion recordings included.
14
+
15
+ ```
16
+ ┌─────────────────────────────────────────────┐
17
+ awwwards.com ─────►│ src/awwwards.ts HTTP client (1 req/s, │
18
+ │ BlockedError on 403/429) │
19
+ └───────────────┬─────────────────────────────┘
20
+ │ HTML
21
+ ┌───────────────▼───────────────┐
22
+ │ src/parsers.ts regex parsers │
23
+ │ parseListing / parseDetail / │
24
+ │ parseJuryDimensions / │
25
+ │ parseCategories / parseElements
26
+ └───────────────┬───────────────┘
27
+ │ rows
28
+ ┌───────────────────────────▼───────────────────────────┐
29
+ │ src/cache.ts SQLite (node:sqlite) at ~/.awwwards-mcp │
30
+ │ sites table + FTS5 (probe-guarded, derived) + asset │
31
+ │ image cache (sha1 of CDN path, no TTL) │
32
+ └───────────────────────────┬───────────────────────────┘
33
+ │
34
+ ┌───────────────────────────▼───────────────────────────┐
35
+ │ src/server.ts 7 MCP tools over createHandlers(deps) │
36
+ └───────────────────────────┬───────────────────────────┘
37
+ │
38
+ src/cli.ts ── stdio McpServer (awwwards-mcp)
39
+ src/index-cli.ts ── crawl CLI (awwwards-index)
40
+ ```
41
+
42
+ ## Module map (`src/`)
43
+
44
+ | Module | Lines | Role |
45
+ |---|---|---|
46
+ | `awwwards.ts` | ~134 | HTTP client. `BASE_URL` / `ASSETS_URL`, `AWARD_FILTERS`, `buildFilterUrl`, `RateLimiter` (promise-chained), `AwwwardsClient` with one retry on transient failures — **never** retries 403/429 (throws `BlockedError`). All binary CDN assets flow through one `getAsset` path. |
47
+ | `parsers.ts` | ~250 | Dependency-free regex parsers. See [parsers & drift](#parsers--drift-the-safety-net) below. |
48
+ | `cache.ts` | ~262 | SQLite storage. FTS5 layer is **derived and probe-guarded**: if the Node build lacks FTS5, `searchSites` returns null and callers fall back to substring search. Per-operation DB open/close (an open handle makes cache files undeletable on Windows). Batched-transaction upserts (per-insert autocommit pays ~4 ms disk sync on Windows). |
49
+ | `indexer.ts` | ~153 | `runIndexer`: meta-keyed lock (`index:lock`, 30 min stale), incremental crawl via `index:progress`, abort-safe status writes. `isIndexStale` gates the 7-day background re-index. |
50
+ | `server.ts` | ~700 | All MCP tool handlers + search merge semantics. The biggest and most rule-dense module — see [search semantics](#search_sites-semantics) before touching it. |
51
+ | `motion.ts` | ~429 | `recordSiteMotion`: webm recording + ffmpeg filmstrip under a strict 30 s budget (documented timing table at the top of the file). Virtual SVG cursor, per-call `mkdtemp` isolation, 10-min stale-tmp sweep, 60 s ffmpeg deadline. |
52
+ | `structure.ts` | ~249 | Band map. `SCAN_SNIPPET` runs in-page via `page.evaluate` (self-contained, reaches DOM through `globalThis` — the Node tsconfig has no DOM lib). `collapseBands` is pure: 8 px sweep, smallest-covering candidate wins, sliver absorb, max-40 cap. Owns the ONE `preScroll` shared with capture. |
53
+ | `capture.ts` | ~66 | Full-page screenshot. Intentionally circular import with `structure.ts` (call-time-only bindings; ESM resolves it). |
54
+ | `cli.ts` | ~179 | `McpServer` wiring, zod schemas, lazy playwright/ffmpeg defaults (optional deps are only touched when a tool actually runs), background auto-reindex, daily version check — stdout stays clean for JSON-RPC. |
55
+ | `version-check.ts` | ~131 | Daily npm-registry check, stderr-only, 3 s timeout, all failures swallowed. Self-update is opt-in via `AWWWARDS_AUTO_UPDATE=1`. |
56
+ | `viewport.ts` | ~20 | Desktop (1440×900) / mobile (390×844, DPR 3, touch) profiles. Width/height go in playwright's `viewport` key; mobile flags spread as sibling context options. |
57
+ | `types.ts`, `index-cli.ts` | — | Shared types (`SiteSummary`, `SiteDetails`, `ElementMedia`); thin crawl-CLI entry. |
58
+
59
+ ## Search semantics (read before touching `server.ts`)
60
+
61
+ `search_sites` invariants, each pinned by tests in `test/server.test.ts`:
62
+
63
+ - **One filter per URL.** Combined filter URLs return 404 on awwwards.com, so
64
+ `buildFilterUrl` picks the highest-priority filter (color > award >
65
+ technology > first tag) and the rest are verified client-side by
66
+ `matchesFilters`. `honorUrlSource` skips re-checking the URL filter for
67
+ freshly scraped rows only.
68
+ - **FTS skip-query-check.** FTS matches prefix+porter-stem; re-checking the
69
+ query with substring semantics would wrongly drop stem matches
70
+ ("magazines" → Magazine). FTS-sourced rows skip the query check.
71
+ - **Color is never served from cache.** Site rows carry no colors, so a color
72
+ search always scrapes its filter page.
73
+ - **Page-window merge.** An empty requested window → scrape replaces. A
74
+ *partially* filled window → top-up scrape MERGES: dedupe by slug, verified
75
+ cache rows win duplicates, bm25 rank order is preserved, scraped-only rows
76
+ append newest-first.
77
+ - **Top-up scrapes are best-effort.** A failed fetch must not escape to the
78
+ stale-fallback catch, which would silently discard the cached page window.
79
+ - **Never cache an all-empty parse.** `get_site_details` /
80
+ `get_site_elements` treat an all-empty detail as layout drift and leave it
81
+ uncached so the mismatch keeps surfacing. One fetch feeds both caches.
82
+ - **Zero results get hints, not live fetches.** Loose-match OR hint (up to 3
83
+ slugs) + taxonomy suggestions computed from cached categories only.
84
+ - **Stale fallback.** On live-request failure, serve stale cache — and the
85
+ cache lookup itself is guarded (the store may be the failure source).
86
+
87
+ ## Parsers & drift: the safety net
88
+
89
+ awwwards.com is a moving target, so the parsers follow strict conventions:
90
+
91
+ - Every regex / `split` / `indexOf` anchor carries a date-stamped
92
+ verification note ("verified live 2026-09-XX").
93
+ - **`parseJuryDimensions` refuses to guess**: if the chartbar labels are not
94
+ exactly Design/Usability/Creativity/Content, it returns undefined instead
95
+ of mapping by position.
96
+ - `parseElements`: `null` = no Elements section (legitimate); empty array =
97
+ markup drift (never cached).
98
+ - Description: curated `>Description</h2>` section primary (~16 % coverage),
99
+ `og:description` meta fallback (~50 %).
100
+ - Elements media paths come in two schemes — modern `element/YYYY/MM/…` and
101
+ 2018-era `external/YYYY/MM/…` — both with `_static.jpeg` posters on the CDN.
102
+
103
+ Three layers guard these anchors:
104
+
105
+ 1. **Fixtures** — real saved awwwards pages in `test/fixtures/` (listing +
106
+ four detail variants, 300–600 KB each). All parser tests are offline.
107
+ 2. **Drift probe** — `scripts/parser-drift-probe.mjs` (`npm run drift`)
108
+ checks every parser anchor against the live listing and detail pages
109
+ (2 fetches at 1 req/s) and against the committed fixtures
110
+ (`npm run drift -- --fixture`, offline). Writes `.drift/status.json`.
111
+ 3. **CI** — a daily GitHub Action (`.github/workflows/parser-drift.yml`)
112
+ runs the live probe and opens/updates a single tracking issue on drift
113
+ (auto-closes when green).
114
+
115
+ When a drift issue lands, the fresh HTML snapshot becomes the new fixture.
116
+
117
+ ## Tests
118
+
119
+ ```bash
120
+ npm test # offline, vitest, fixtures only — no network
121
+ npm run smoke # manual live smoke test (test/live-smoke.ts)
122
+ npm run build # tsc → dist/
123
+ ```
124
+
125
+ Suites: `parsers` (fixtures pin every anchor), `cache` (FTS on/off, TTLs),
126
+ `server` (the merge semantics above — the largest suite), `indexer` (lock,
127
+ resume, abort), `motion` / `capture` / `structure` (fake playwright +
128
+ injectable ffmpeg — nothing launches a browser), `awwwards` (fetch fake),
129
+ `version-check`. `vitest.config.ts` raises the timeout to 30 s for the
130
+ multi-page indexer crawls.
131
+
132
+ Dependency injection is the pattern everywhere a hard dependency exists:
133
+ `AwwwardsClient.fetchFn`, `Cache` paths + clock, playwright loaders,
134
+ `ffmpegFn`. Write new tests against fakes, not networks.
135
+
136
+ ## Scripts & tooling
137
+
138
+ | Script | Purpose |
139
+ |---|---|
140
+ | `scripts/doctor.mjs` | `npm run doctor` — checks network, drift, deps, cache, boot; applies fixes. |
141
+ | `scripts/parser-drift-probe.mjs` | `npm run drift` — live + fixture anchor probe (see above). |
142
+ | `scripts/enrich-styles.mjs` | Phase-1 style-classification experiment via a self-hosted simple-jev classifier (spec: `docs/superpowers/specs/2026-09-20-jev-style-enrichment-design.md`). |
143
+ | `scripts/make-demo.mjs` | Builds the README demo assets. |
144
+ | `scripts/record-scrollthrough.mjs` | The validated recording script `motion.ts` was ported from. |
145
+ | `scripts/skill-memory.mjs` | Journal → techniques distillation for the skills flywheel (`skills/_memory/techniques.json`). |
146
+
147
+ ## Skills & showcase
148
+
149
+ - `skills/` ships three agent skills (`awwwards-inspiration`,
150
+ `awwwards-motion-study`, `awwwards-doctor`). Shipped copies change only via
151
+ human PRs; local copies self-improve through the skill-memory flywheel.
152
+ - `showcase/` holds real sites built through the loop (`ridge`,
153
+ `afjal-portfolio`, `fallow-press`), each with `_qa/` capture evidence. They
154
+ double as regression demos for the capture/structure/motion tools.
155
+
156
+ ## Conventions worth preserving
157
+
158
+ These are hard-won; check for them before "simplifying":
159
+
160
+ - `page.evaluate` snippets are self-contained and reach browser globals via
161
+ `globalThis` (no DOM lib in tsconfig).
162
+ - Per-operation SQLite open/close; batched transaction upserts.
163
+ - Optional heavy deps (playwright, ffmpeg-static) stay unresolved at compile
164
+ time and resolve lazily at runtime with in-band install hints.
165
+ - stdout is the MCP JSON-RPC channel — every log/notice goes to stderr.
166
+ - New crawler code must keep the politeness contract: `/websites/…` paths
167
+ only, 1 req/s, no query-string pagination (`/tag/`, `/search-websites`,
168
+ `/elements/*` etc. are robots-disallowed and never fetched).
169
+
170
+ ## Contributing
171
+
172
+ PRs welcome — parser-drift fixes especially (a fresh HTML snapshot attached
173
+ to a drift issue is the fastest merged PR). See [CONTRIBUTING.md](CONTRIBUTING.md)
174
+ for branch naming, the PR template and the politeness constraints. Security
175
+ issues go privately via [SECURITY.md](SECURITY.md).
176
+
177
+ MIT — see [LICENSE](LICENSE).
package/README.md CHANGED
@@ -18,17 +18,29 @@ queries keep AND semantics: every token must hit the same site.
18
18
 
19
19
  | Tool | What it does |
20
20
  |------|--------------|
21
- | `search_sites` | Search by color, tags, technology, award type or free-text query. Multi-word queries match against the local FTS5 index and rank BM25 (title hits lead); zero results come with loose-match and taxonomy-tag hints. Returns site cards with inline screenshots. |
21
+ | `search_sites` | Search by color, tags, technology, award type or free-text query. Multi-word queries match against the local FTS5 index and rank BM25 (title hits lead); zero results come with loose-match and taxonomy-tag hints. Default full results include each site's screenshot; use `responseMode: "compact"` for concise cards and inline previews of only the first two results on the requested page. |
22
22
  | `get_site_details` | Full design DNA for one site: palette, technologies, elements, awards, description. |
23
+ | `compare_sites` | Compare 2–3 sites' design DNA and jury scores as text-only JSON. Uses cached details or fetches missing detail pages. |
24
+ | `get_index_status` | Read local index count, crawl progress, last success/error, lock state and freshness without network requests. |
23
25
  | `get_site_elements` | Component-level visuals for one site: each element's poster image inline (3D models, video content, mobile layouts, microcopy…) + video URLs. |
24
26
  | `list_categories` | Every filter the agent can search by (200+ tags, 27 colors). |
25
27
  | `capture_live_site` | Optional: fresh full-page screenshot of any live URL. Waits for `load` + a settle window with a bounded pre-scroll, so heavy sites work (`waitStrategy: "networkidle"` available). Pass `viewport: "mobile"` for the 390×844 iPhone-class render (`"desktop"` 1440×900 default). (needs [playwright](https://playwright.dev)). |
26
28
  | `analyze_page_structure` | Section band map of any page (live URL or local file:// build): tag, background, offset, height per band. Compare a reference site's structure against your build. Same heavy-site-friendly wait (`waitStrategy: "networkidle"` available); `viewport: "mobile"` analyzes the phone-class layout (`"desktop"` default). (needs [playwright](https://playwright.dev)). |
27
29
  | `record_site_motion` | Optional: short motion-through video of a live URL — preloader, scroll-triggered and hover/cursor animations. Returns an inline filmstrip JPEG plus the saved .webm path. `viewport: "mobile"` records at phone size — the filmstrip renders at the selected viewport, no pillarboxing (`"desktop"` default). (needs [playwright](https://playwright.dev) + ffmpeg-static). |
30
+ | `search_elements` | Search the inspiration-elements gallery (footer, hero, pricing, 404…) by free text; ranks BM25 over title/author/category. |
31
+ | `get_element` | One element record: title, category, author, built-with stack, related elements and its media URL (image or video) pointing at awwwards' CDN. |
32
+ | `get_motion_dna` | Runtime motion fingerprint of a live URL: animation libraries, render engines, ScrollTrigger stats (trigger count, scrub ratio), tween easing/duration vocab and the scroll model. Fresh capture or cached capture with timestamp. |
33
+ | `search_motion` | Search previously captured motion-DNA scans by library, scroll model or easing vocabulary — find references by how a site moves. |
34
+ | `new_winners` | Poll today's freshly-crowned winners (SOTD / Developer Award / Honorable Mention) against a persisted baseline. First call seeds and dumps the listing; later calls report the delta. Each first-seen winner's Elements section is backfilled into the searchable element corpus, so new winners are element-searchable immediately. |
35
+ | `watch_site` | Persistent watches over studios, tags or specific sites (add/list/remove). `list` matches each watch against the freshest cached listing and reports per-watch `NEW since last check` deltas — it never fetches; `new_winners`/`search_sites` keep the pool fresh. |
36
+
37
+ **Data posture**: element records store metadata + media URLs pointing at
38
+ awwwards' own CDN — nothing is mirrored. Motion DNA records are local
39
+ captures, each stamped with the time it was taken.
28
40
 
29
41
  ## Setup
30
42
 
31
- Any MCP-compatible coding agent can use awwwards-mcp — no API key, no account.
43
+ **v1.0.0** — the first stable release. Any MCP-compatible coding agent can use awwwards-mcp — no API key, no account.
32
44
  Requires Node ≥ 22.13 (`node -v` to check). Pick your agent:
33
45
 
34
46
  **Updates**: the server checks the npm registry once a day and prints an
@@ -124,18 +136,19 @@ package automatically if present.
124
136
 
125
137
  ## Skills
126
138
 
127
- This package ships three agent skills. Any agent that follows the
139
+ This package ships four agent skills. Any agent that follows the
128
140
  [Agent Skills standard](https://agentskills.io) can load them; copy them into
129
141
  your agent's skills directory:
130
142
 
131
143
  ```bash
132
144
  npm install awwwards-mcp
133
- mkdir -p ~/.agents/skills && cp -r node_modules/awwwards-mcp/skills/awwwards-inspiration node_modules/awwwards-mcp/skills/awwwards-doctor node_modules/awwwards-mcp/skills/awwwards-motion-study ~/.agents/skills/
145
+ mkdir -p ~/.agents/skills && cp -r node_modules/awwwards-mcp/skills/awwwards-inspiration node_modules/awwwards-mcp/skills/awwwards-setup node_modules/awwwards-mcp/skills/awwwards-doctor node_modules/awwwards-mcp/skills/awwwards-motion-study ~/.agents/skills/
134
146
  ```
135
147
 
136
148
  | Skill | What it teaches |
137
149
  |-------|-----------------|
138
- | `awwwards-inspiration` | The inspiration loop: search, judge from screenshots, pull design DNA, state a design direction, capture/motion-first builds. |
150
+ | `awwwards-setup` | First-time onboarding: asks the user's preferences (result density, viewport, captures, local index, winner watches), persists them to `~/.awwwards-mcp/preferences.json`, and runs any one-time installs they opt into. |
151
+ | `awwwards-inspiration` | The inspiration loop: search, judge from screenshots, pull design DNA, state a design direction, capture/motion-first builds; staying current with `new_winners` and `watch_site`. |
139
152
  | `awwwards-motion-study` | The full video chain: what to record from a live site (and what to skip), frame-by-frame review (video input or tile-per-element), the motion inventory, and build verification by re-recording. |
140
153
  | `awwwards-doctor` | Repair: run `npm run doctor`, apply its fixes, re-anchor parsers after real awwwards.com drift, recover the in-flight task that surfaced the failure. |
141
154
 
@@ -147,7 +160,7 @@ mkdir -p ~/.agents/skills && cp -r node_modules/awwwards-mcp/skills/awwwards-ins
147
160
  | Agent Skills-standard agents | `~/.agents/skills/` |
148
161
 
149
162
  Windows: run this from Git Bash, or copy
150
- `node_modules\awwwards-mcp\skills\awwwards-inspiration` manually.
163
+ `node_modules\awwwards-mcp\skills\awwwards-setup` manually.
151
164
 
152
165
  ## Indexing (recommended)
153
166
 
@@ -165,10 +178,43 @@ npm run index
165
178
  SQLite cache at `~/.awwwards-mcp/`.
166
179
  - Resumable: interrupt it and re-run — completed pages are skipped.
167
180
  - The MCP server re-indexes automatically in the background whenever the
168
- index is older than 7 days (never blocking your session).
181
+ index is older than 7 days (never blocking your session). Completed crawl
182
+ checkpoints are cleared so each refresh actually revisits the tag pages.
183
+ Run `get_index_status` to inspect progress or the last crawl error.
184
+
185
+ ### Elements index (optional)
186
+
187
+ The `search_elements` tool auto-indexes the first gallery page (~48 items)
188
+ on first use. To build a full corpus (~1,500+ items, ~15 pages at 48/page):
189
+
190
+ ```bash
191
+ npm run index -- --elements all # follow pagination until exhausted
192
+ npm run index -- --elements 10 # first 10 listing pages
193
+ ```
194
+
195
+ Any crawl beyond the first page also runs the **taxonomy pass**: every
196
+ facet page (`/elements/footer/`, `/elements/cta/`, … 46 categories) is
197
+ fetched once and each element under it is stamped with that category, so
198
+ `search_elements`' `category` filter works across the corpus. Element
199
+ pages carry no breadcrumb, so this listing-side pass is the only category
200
+ source; elements seen on no facet page stay `unsorted`.
201
+
202
+ Element rows are searched by title, author, category **and slug tokens**
203
+ (slug is an FTS5-indexed column; a cache opened from an older schema
204
+ version rebuilds its search index automatically on first open). Each
205
+ element also carries the slug of the award-winning site it came from
206
+ (`siteSlug`/`siteUrl` in `search_elements`/`get_element` results). Two
207
+ record sources share this corpus: gallery records (indexed from the public
208
+ elements listing) and `source:"site"` records backfilled from each new
209
+ SOTD winner's own Elements section by `new_winners` — their slugs are
210
+ namespaced `site-<siteslug>-<title>` so the two never collide. The
211
+ elements index has the same 7-day freshness gate as the sites index —
212
+ a re-run inside the window skips itself.
169
213
 
170
214
  Site details (palettes, tech stacks) are still fetched on demand and cached
171
- for 7 days.
215
+ for 7 days. Awwwards page and CDN requests have a 10-second deadline per
216
+ attempt, including response-body reading; transient page failures are retried
217
+ once, while blocks (403/429) and CDN failures are not retried.
172
218
 
173
219
  ## How it works
174
220
 
@@ -203,9 +249,23 @@ skill.
203
249
  > results would be better still. The skill's frame-tile doctrine is what
204
250
  > closes that gap today.
205
251
 
252
+ **Watch the whole loop run (1:50):**
253
+
254
+ <video src="assets/demo-loop.mp4" controls muted playsinline></video>
255
+
256
+ *Screen recording of the agent running the `awwwards-inspiration` loop end to
257
+ end with the awwwards MCP tools — searching SOTD references with inline
258
+ screenshots, pulling design DNA, frame-studying element videos, building, and
259
+ verifying with band maps + motion recording. If your client doesn't render
260
+ the player, [watch the file directly](assets/demo-loop.mp4).*
261
+
206
262
  **1. [Ridge](showcase/ridge/index.html)**
207
263
  ([source](showcase/ridge/)) — a Swiss-minimal single-page showcase for a fictional engineering-talent studio, direction **Aspen Search** (SOTD + Developer Award, jury 7.48): monochrome `#FAFAF8`/`#1A1A1A` + mint, giant grotesque section markers, halftone grain, asymmetric panel grid, dark discipline panels in an interior **horizontal pin passage**, count-up stats, client rows, theme toggle, cursor-follower. Built with the v1.4.0 toolkit: FTS5-ranked direction search, **both-viewports** reference captures and QA (desktop 7,849px + mobile 390×844), overflow audit (0px both), pin-center shots, film verification — and the skill-memory flywheel recorded the findings. QA evidence: `showcase/ridge/_qa/`.
208
264
 
265
+ **The grain-panel hover**, studied from aspensearch.com's recording and rebuilt as a canvas dither-dissolve — dots flip to mint around the mouse, the trail elongates, the boundary dissolves:
266
+
267
+ ![Ridge grain-panel hover — canvas dither-dissolve following the mouse](assets/ridge-dither.gif)
268
+
209
269
  | Panel grid (desktop) | Horizontal discipline passage | Mobile 390×844 |
210
270
  |---|---|---|
211
271
  | ![Ridge desktop — Swiss panel grid with mint and grain](assets/ridge-home.jpg) | ![Ridge disciplines — pinned horizontal passage mid-slide](assets/ridge-disciplines.jpg) | ![Ridge mobile — stacked grid, zero overflow](assets/ridge-mobile.jpg) |
package/dist/awwwards.js CHANGED
@@ -63,9 +63,35 @@ export class RateLimiter {
63
63
  export class AwwwardsClient {
64
64
  rateLimiter;
65
65
  fetchFn;
66
+ timeoutMs;
66
67
  constructor(opts = {}) {
67
68
  this.rateLimiter = opts.rateLimiter ?? new RateLimiter(1000);
68
69
  this.fetchFn = opts.fetchFn ?? fetch;
70
+ this.timeoutMs = opts.timeoutMs ?? 10_000;
71
+ if (!Number.isSafeInteger(this.timeoutMs) || this.timeoutMs < 1 || this.timeoutMs > 2_147_483_647) {
72
+ throw new RangeError("timeoutMs must be a positive integer within the timer range");
73
+ }
74
+ }
75
+ async fetchWithTimeout(url, label, init, read) {
76
+ const controller = new AbortController();
77
+ let timer;
78
+ const timeout = new Promise((_, reject) => {
79
+ timer = setTimeout(() => {
80
+ reject(new Error(`Timed out fetching ${label} from ${url} after ${this.timeoutMs}ms ` +
81
+ "(including response body); check connectivity and try again later."));
82
+ controller.abort();
83
+ }, this.timeoutMs);
84
+ });
85
+ try {
86
+ return await Promise.race([
87
+ (async () => read(await this.fetchFn(url, { ...init, signal: controller.signal })))(),
88
+ timeout,
89
+ ]);
90
+ }
91
+ finally {
92
+ if (timer !== undefined)
93
+ clearTimeout(timer);
94
+ }
69
95
  }
70
96
  // Rate-limited page fetch with one retry on transient failures.
71
97
  // Blocks (403/429) are never retried.
@@ -75,15 +101,13 @@ export class AwwwardsClient {
75
101
  for (let attempt = 0; attempt < 2; attempt++) {
76
102
  try {
77
103
  await this.rateLimiter.acquire();
78
- const res = await this.fetchFn(url, {
79
- headers: { "User-Agent": USER_AGENT },
80
- redirect: "follow",
104
+ return await this.fetchWithTimeout(url, "page", { headers: { "User-Agent": USER_AGENT }, redirect: "follow" }, async (res) => {
105
+ if (res.status === 403 || res.status === 429)
106
+ throw new BlockedError(url, res.status);
107
+ if (!res.ok)
108
+ throw new Error(`HTTP ${res.status} for ${url}`);
109
+ return res.text();
81
110
  });
82
- if (res.status === 403 || res.status === 429)
83
- throw new BlockedError(url, res.status);
84
- if (!res.ok)
85
- throw new Error(`HTTP ${res.status} for ${url}`);
86
- return await res.text();
87
111
  }
88
112
  catch (err) {
89
113
  if (err instanceof BlockedError)
@@ -103,11 +127,10 @@ export class AwwwardsClient {
103
127
  }
104
128
  // Binary CDN assets (thumbnails, element media) all fetch through this single path.
105
129
  async fetchCdnBinary(url, label) {
106
- const res = await this.fetchFn(url, {
107
- headers: { "User-Agent": USER_AGENT },
130
+ return this.fetchWithTimeout(url, label, { headers: { "User-Agent": USER_AGENT } }, async (res) => {
131
+ if (!res.ok)
132
+ throw new Error(`HTTP ${res.status} fetching ${label}`);
133
+ return Buffer.from(await res.arrayBuffer());
108
134
  });
109
- if (!res.ok)
110
- throw new Error(`HTTP ${res.status} fetching ${label}`);
111
- return Buffer.from(await res.arrayBuffer());
112
135
  }
113
136
  }