playlist-data-engine 1.7.2 → 1.8.0

package/docs/features/GATEWAY_RESOLUTION.md

@@ -0,0 +1,725 @@
+# Arweave Gateway Resolution & the AR.IO Wayfinder
+
+## Overview
+
+The playlist-data-engine loads ML models, playlist metadata, artwork, and WASM modules from the Arweave permaweb. Since any single gateway can be slow, down, or return errors, the engine uses a custom **`ArweaveGatewayManager`** combined with **@ar.io/wayfinder-core** (v2.0.1) to provide resilient, multi-tier gateway routing.
+
+Playlists, ML models, audio, and images are all stored on Arweave. Every Arweave URL that the engine encounters passes through this resolution pipeline.
+
+---
+
+## Architecture
+
+```
+resolveUrl(arweaveUrl, { bypassArweaveNet?, bypassWayfinder? })
+│
+├─ 0. Cache hit? ─────────────────────────────────── return immediately
+│
+├─ 1. Persisted gateway (from localStorage — known-working gateway from previous session)
+│
+├─ 2. arweave.net (official gateway — reliable fallback)
+│      └─ Skipped when bypassArweaveNet: true
+│
+├─ 3. Static fallback gateways (ardrive.net, turbo-gateway.com)
+│   └─ Filtered by health data (>70% failure rate skipped after 3+ checks)
+│   └─ All tried in parallel via Promise.any()
+│
+└─ 4. AR.IO Wayfinder (wider pool via composite routing, up to 3 retries)
+       └─ Skipped when bypassWayfinder: true
+       ├─ FastestPingRoutingStrategy (top 10 by operator stake, 3s timeout)
+       └─ RandomRoutingStrategy (top 20 by operator stake, fallback)
+```
+
+Resolution is **sequential between tiers** but **parallel within the fallback tier** — each tier is tried in order, and within the static fallback tier (step 3), all gateways are checked in parallel via `Promise.any()`. The first gateway that passes a real HEAD check wins; remaining in-flight checks are aborted. Non-Arweave URLs pass through unchanged.
+
+If the URL returned by `resolveUrl()` actually fails during data transfer, call [`reportGatewayFailure()`](#reportgatewayfailureurl-options) to trigger a fresh resolution that **excludes the failed gateway**:
+
+```
+reportGatewayFailure() retry order:
+1. arweave.net (if not the failed host)
+2. Wayfinder
+3. Remaining fallback gateways (excluding the failed host)
+```
+
+---
+
+## The Resolution Pipeline
+
+When [`resolveUrl()`](#resolveurlurl-signal) is called with an Arweave URL, the manager follows this fallback chain:
+
+```
+0. Cache lookup (per-txId, 24-hour TTL) — return immediately if hit
+1. Persisted gateway from localStorage (known-working gateway from previous session)
+2. arweave.net (official gateway — reliable fallback)
+3. Remaining static fallback gateways (ardrive.net → turbo-gateway.com, parallel)
+4. AR.IO Wayfinder (dynamic routing via NetworkGatewaysProvider, up to 3 retries)
+```
+
+Each step runs a real `HEAD` request to verify the gateway can serve the specific transaction before returning it. If the signal is already aborted when called, the method returns immediately without doing any work.
+
+**Request deduplication**: If multiple callers request the same uncached `txId` concurrently, only one runs the full fallback chain — others share its result via an in-flight promise map. The promise is cleaned up once it resolves or rejects.
+
+---
+
+## AR.IO Wayfinder Integration
+
+### What Wayfinder Is
+
+AR.IO Wayfinder is a dynamic gateway routing system for Arweave. Instead of relying on a hardcoded list of gateways, it selects the best gateway from a network of community-operated Arweave gateways ranked by operator stake.
+
+The engine uses `@ar.io/wayfinder-core` (v2.0.1) and `@ar.io/sdk` (v4.0.3). Both ship as **regular dependencies** — they install with the engine rather than being peer dependencies the consumer must provide — and are **externalized** in the Vite build, so the library imports them at runtime instead of bundling them.
+
+### Lazy Dynamic Import
+
+Wayfinder and the AR.IO SDK are loaded asynchronously via separate dynamic `import()` calls to avoid pulling Node.js `crypto` polyfills into the main bundle:
+
+```typescript
+// src/utils/arweaveGatewayManager.ts:38-49
+async function loadWayfinder(): Promise<typeof import('@ar.io/wayfinder-core') | null> {
+    try {
+        return await import('@ar.io/wayfinder-core');
+    } catch {
+        return null;
+    }
+}
+
+async function loadArioSdk(): Promise<typeof import('@ar.io/sdk') | null> {
+    try {
+        return await import('@ar.io/sdk');
+    } catch {
+        return null;
+    }
+}
+```
+
+If either package fails to load (missing dependency, browser environment without support, etc.), the manager degrades gracefully — static gateway fallback continues to work.
+
+### Three Initialization Tiers
+
+Both packages must load successfully for the full composite routing strategy. The manager handles three initialization tiers:
+
+1. **Full mode**: Both `@ar.io/wayfinder-core` and `@ar.io/sdk` available → custom CompositeRoutingStrategy (FastestPing + Random)
+2. **Basic mode**: Only `@ar.io/wayfinder-core` available → default Wayfinder routing (no custom strategy)
+3. **Offline mode**: Neither available → static gateways only
+
+### Composite Routing Strategy
+
+When both packages load, the Wayfinder client is initialized with a two-layer routing strategy in [arweaveGatewayManager.ts:338-383](../../src/utils/arweaveGatewayManager.ts#L338-L383):
+
+```typescript
+const rpc = solanaKit.createSolanaRpc(this.solanaRpcUrl);
+const arioClient = ARIO.init({ rpc });
+
+const primaryProvider = new NetworkGatewaysProvider({
+    ario: arioClient,
+    sortBy: 'weights.compositeWeight',
+    limit: 50,
+});
+
+const fallbackProvider = new NetworkGatewaysProvider({
+    ario: arioClient,
+    sortBy: 'weights.compositeWeight',
+    limit: 100,
+});
+
+this.wayfinder = wfMod.createWayfinderClient({
+    routingStrategy: new CompositeRoutingStrategy({
+        strategies: [
+            new FastestPingRoutingStrategy({ timeoutMs: 3000, gatewaysProvider: primaryProvider }),
+            new RandomRoutingStrategy({ gatewaysProvider: fallbackProvider }),
+        ],
+    }),
+});
+```
+
+| Strategy | Gateway Pool | Timeout | Purpose |
+|----------|-------------|---------|---------|
+| `FastestPingRoutingStrategy` | Top 50 by composite weight | 3000ms | Primary: ping all 50, pick the fastest |
+| `RandomRoutingStrategy` | Top 100 by composite weight | — | Fallback: if all pings fail, pick randomly from wider pool |
+
+If either `@ar.io/sdk` is unavailable or the custom strategy throws, it falls back to `createWayfinderClient()` with no arguments (default routing).
+
+### Tuning the Gateway Pool
+
+The `NetworkGatewaysProvider` is what decides *which* gateways Wayfinder considers. Two knobs matter: **`sortBy`** (how to rank the network's hundreds of gateways) and **`limit`** (how many to keep). Both are exposed as config on `ArweaveGatewayManager` — see [Configuration Options](#configuration-options) below.
+
+**`wayfinderSortBy` options** (defined in `@ar.io/wayfinder-core`'s `SortBy` type):
+
+| Value | Ranks by | Use when |
+|---|---|---|
+| `weights.compositeWeight` *(default)* | AR.IO's combined performance + tenure + stake + observer score | You want the protocol's own "best overall" ranking. Recommended general default. |
+| `weights.normalizedCompositeWeight` | Composite weight normalized across the network | Similar to compositeWeight but scale-adjusted. |
+| `weights.gatewayPerformanceRatio` | Recent uptime / response performance | You care most about speed and reliability, not network reputation. |
+| `weights.tenureWeight` | Time in good standing | Older = battle-tested, but stale gateways may have decayed. |
+| `weights.stakeWeight` | Normalized operator stake | Same problem as raw stake — surfaces whales. |
+| `stats.passedConsecutiveEpochs` | Consecutive epochs the gateway passed observation | Reliability proxy — gateways that consistently meet network checks. |
+| `totalDelegatedStake` | Stake delegated *by other users* | Weak community-trust signal. |
+| `startTimestamp` | When the gateway joined the network | Roughly correlates with tenure. |
+| `operatorStake` | Raw tokens locked by the operator | **Not recommended** — a single operator can spam subdomains (e.g. `*.noddex.com`) and dominate the top N. |
+
+**`limit`** controls how many gateways the provider returns. The two providers use different limits on purpose:
+- **`wayfinderPrimaryLimit`** (default 50): pool used by `FastestPingRoutingStrategy`. Keep small enough that pinging all of them stays under the 3s timeout.
+- **`wayfinderFallbackLimit`** (default 100): pool used by `RandomRoutingStrategy`. Wider pools cost nothing because `Random` just picks one without pinging — they just give more variety across sessions.
+
+**Why one operator can dominate**: AR.IO ranks per *gateway record*, not per operator. A single operator running `sol01.example.com` through `sol10.example.com` will fill 10 slots if they rank high on the chosen `sortBy`. `compositeWeight` mitigates this somewhat (subdomains generally have similar weights so they cluster together), but the only real fix is custom base-domain deduplication wrapping the provider — not currently implemented.
+
+### Wayfinder Strategy Presets
+
+The `wayfinderStrategy` config option picks the routing approach. Presets map to combinations of `@ar.io/wayfinder-core` strategy classes so consumers don't have to import the SDK themselves.
+
+| Preset | Behavior | Use when |
+|---|---|---|
+| `'composite-ping-random'` *(default)* | `FastestPing` over primary pool → falls back to `Random` over fallback pool | Balanced speed + resilience. Recommended default. |
+| `'random-only'` | Pick uniformly at random from fallback pool, no pinging | You want max variety across sessions and don't care about ping-optimal selection. Spreads load across the network. |
+| `'ping-only'` | Ping primary pool, pick fastest, no random fallback | You want strict "fastest available" behavior. If all pings fail, Wayfinder returns nothing and static fallbacks take over. |
+| `'round-robin'` | Cycle through primary pool in order | Predictable distribution. No health awareness — a dead gateway in the rotation will be picked. |
+
+Strategy classes Wayfinder also exposes but aren't covered by presets (would require forking): `StaticRoutingStrategy` (always one hardcoded gateway), `PreferredWithFallbackRoutingStrategy` (try a pinned gateway first), and raw `CompositeRoutingStrategy` with custom chain order. Add a new preset value to [arweaveGatewayManager.ts](../../src/utils/arweaveGatewayManager.ts) if you need one of these.
+
+### Configuration Options
+
+All Wayfinder knobs are passable to the `ArweaveGatewayManager` constructor:
+
+```typescript
+import { ArweaveGatewayManager } from 'playlist-data-engine';
+
+const manager = new ArweaveGatewayManager({
+    // Solana RPC for ar.io SDK registry reads
+    solanaRpcUrl: 'https://mainnet.helius-rpc.com/?api-key=YOUR_KEY',
+
+    // How AR.IO ranks gateways before building the candidate pool
+    wayfinderSortBy: 'weights.gatewayPerformanceRatio',
+
+    // Pool sizes
+    wayfinderPrimaryLimit: 30,    // FastestPing pool (default 50)
+    wayfinderFallbackLimit: 200,  // Random pool (default 100)
+
+    // Routing strategy preset
+    wayfinderStrategy: 'random-only',
+});
+```
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `solanaRpcUrl` | `string` | `https://solana-rpc.publicnode.com` | Solana RPC for `@ar.io/sdk` registry reads. |
+| `wayfinderSortBy` | `WayfinderSortBy` | `'weights.compositeWeight'` | Gateway ranking field — see table above. |
+| `wayfinderPrimaryLimit` | `number` | `50` | Primary (FastestPing) pool size. |
+| `wayfinderFallbackLimit` | `number` | `100` | Fallback (Random) pool size. |
+| `wayfinderStrategy` | `WayfinderStrategy` | `'composite-ping-random'` | Routing strategy preset — see table above. |
+
+### Solana RPC Dependency
+
+As of `@ar.io/sdk` v4, the AR.IO gateway registry lives on Solana — `ARIO.init({ rpc })` requires a Solana RPC client to read the gateway list that powers `NetworkGatewaysProvider`. Without a working Solana RPC, `NetworkGatewaysProvider` returns no gateways and `CompositeRoutingStrategy` fails with `all strategies failed`, killing the Wayfinder fallback entirely (static gateways still work).
+
+The RPC URL is configurable via `ArweaveGatewayManagerConfig.solanaRpcUrl`:
+
+```typescript
+new ArweaveGatewayManager({
+    solanaRpcUrl: 'https://mainnet.helius-rpc.com/?api-key=YOUR_KEY',
+});
+```
+
+**Default:** `https://solana-rpc.publicnode.com` — public, CORS-enabled, no signup required. Works out of the box but is rate-limited; override for production.
+
+**Do not use `https://api.mainnet-beta.solana.com`** — Solana Labs blocks browser-origin requests with HTTP 403, which guarantees Wayfinder will fail in any browser context.
+
+**Recommended providers for production:** Helius, QuickNode, Triton, Alchemy — all have free tiers with CORS enabled and much higher rate limits than the public RPC.
+
+**Default behavior changes based on whether you provide a custom RPC:** When `solanaRpcUrl` is not set, the manager treats Wayfinder as opt-in — `resolveUrl(url)` defaults to `bypassWayfinder: true`. To use Wayfinder with the default RPC, you must explicitly pass `resolveUrl(url, { bypassWayfinder: false })`. When you do provide a custom `solanaRpcUrl`, Wayfinder is opt-out — it runs unless you pass `bypassWayfinder: true`. This avoids silently degrading resolution when the rate-limited public RPC fails.
+
+### How Wayfinder Is Used
+
+Wayfinder is **not the primary resolution path**. It sits after static fallbacks as a last resort in [arweaveGatewayManager.ts:599-610](../../src/utils/arweaveGatewayManager.ts#L599-L610):
+
+```typescript
+// Step 4: Try Wayfinder as last resort (wider pool, but slower)
+if (this.wayfinder) {
+    this.logger.debug('Static gateways failed, trying Wayfinder', { txId });
+    const wayfinderResult = await this.tryWayfinder(url, txId, pathSuffix, signal);
+    if (wayfinderResult) return wayfinderResult;
+}
+```
+
+The `tryWayfinder()` method uses a **first-pick + walk-the-list** strategy:
+
+1. **First pick** — calls `this.wayfinder.resolveUrl({ originalUrl: url })` with a 7-second timeout. This lets Wayfinder's configured routing strategy (e.g. `FastestPingRoutingStrategy`) pick the best candidate, benefiting from its internal ping caching.
+2. Verifies the first pick with a real HEAD check against the txId. If it passes, sets it as the active gateway and returns.
+3. **Walk the ranked list** — if the first pick fails verification, instead of asking Wayfinder for a new pick (which re-runs the whole ping race), the engine fetches the ranked gateway list directly from `NetworkGatewaysProvider` and walks it in rank order. Up to `MAX_WALK_DEPTH = 10` candidates are HEAD-checked; the first that has the file wins.
+4. Already-failed hosts (from the first pick and any prior chain steps) are skipped during the walk.
+
+Key design choices:
+- **Wayfinder picks aren't trusted blindly** — even though Wayfinder has its own health checks, the engine verifies each candidate can serve the specific transaction. Prevents false positives where Wayfinder returns a gateway that works for the network but doesn't have the specific file.
+- **No redundant ping races** — the original retry loop re-ran Wayfinder's full ping selection on each attempt, which often returned the same fastest-but-broken gateway. Walking the ranked list is faster and explores actually different candidates.
+- **Ranked list is cached** — see [Ranked Gateway List Caching](#ranked-gateway-list-caching).
+
+---
+
+## Custom Extensions (Beyond Stock Wayfinder)
+
+### Persisted Gateway State
+
+The active gateway is persisted to `localStorage` under the key `arweave_active_gateway` with a **2-hour TTL**. On the next session or page load, the persisted gateway is restored and tried first (before arweave.net). If it's expired or fails, it's cleared immediately.
+
+This is a custom addition — Wayfinder itself has no persistence. It means the engine remembers which gateway was working across sessions, avoiding the latency of Wayfinder's ping-based routing on every startup.
+
+### Ranked Gateway List Caching
+
+The ranked gateway list returned by `NetworkGatewaysProvider.getGateways()` is cached in two layers to minimize Solana RPC traffic:
+
+1. **In-memory cache** — held on the manager instance after the first fetch. Used for the walk step of `tryWayfinder()` so retries within a single session never re-hit Solana.
+2. **localStorage persistence** — written under the key `arweave_ranked_gateways` whenever the in-memory cache refreshes. On the next page load, the persisted list hydrates the in-memory cache before any Wayfinder call.
+
+Both layers share the same **2-hour TTL** (`RANKED_GATEWAYS_TTL_MS`). The TTL is intentionally long: the AR.IO registry is slow-moving (gateways stake AR tokens to join, rankings shift on epoch-length performance metrics), so the top 50 by `compositeWeight` is essentially stable across hours. When the TTL expires, the next walk triggers a single refresh from the provider, which writes back to both layers. If the refresh fails (e.g. RPC outage), the stale cache is used as a fallback rather than returning nothing.
+
+**Net effect on Solana RPC usage:**
+- Cold start (no cache): 1 RPC call to fetch the list
+- Returning visit within 2 hours: 0 RPC calls (loaded from localStorage)
+- After 2 hours: 1 RPC call to refresh
+- Per file resolve within the cache window: 0 additional RPC calls
+
+This is especially important when using the default public RPC (`solana-rpc.publicnode.com`), since it's rate-limited. Persisting the list across sessions means even heavy users typically only make a handful of RPC calls per hour.
+
+### Slow Gateway Detection & Tracking
+
+The engine tracks real fetch timing (not just HEAD checks) to detect degrading gateways *before* they fail completely:
+
+- **[`reportFetchSuccess(timingMs)`](#reportfetchsuccesstimingms)**: Call after a successful fetch. If the timing exceeds `slowResponseThreshold` (default: 8000ms), a `consecutiveSlowResponses` counter increments. Fast responses reset it.
+- **[`reportFetchTiming(host, timingMs, success)`](#reportfetchtiminghost-timingms-success)**: Lower-level method that records timing data for any gateway (not just the active one), feeding into health tracking.
+
+When `consecutiveSlowResponses` reaches `maxSlowResponses` (default: 3), the **next** `resolveUrl()` call proactively rotates away from the active gateway: it clears the active gateway, the persisted gateway, and the per-txId cache, then re-runs the full fallback chain to find a faster gateway.
+
+### Health-Aware Fallback Filtering
+
+When trying fallback gateways (step 3), the engine filters out gateways with a **>70% failure rate** (and at least 3 recorded checks). This prevents repeatedly trying known-dead gateways. If all gateways would be filtered, it uses the unfiltered list instead (graceful degradation).
+
+This is a custom health tracking system built on top of `ResponseTimeRecord[]` — not a Wayfinder feature. It tracks per gateway:
+- Success/failure counts
+- Average response times
+- Last success/failure timestamps
+- Success rate (healthy = >= 50%)
+
+### AbortSignal Propagation
+
+Every resolution path supports `AbortSignal` for cancellation:
+- `resolveUrl(url, signal?)` — cancel in-flight gateway checks
+- `reportGatewayFailure(url, { signal? })` — cancel retry resolution
+
+If the signal is already aborted when called, the method returns the original URL immediately without doing any work.
+
+### Failure Reason Differentiation
+
+[`reportGatewayFailure()`](#reportgatewayfailureurl-options) accepts a `reason` parameter to distinguish failure types:
+
+| Reason | Behavior |
+|--------|----------|
+| `'load-error'` | Gateway failed — clear active gateway, find new one |
+| `'user-cancel-slow'` | User cancelled after waiting — gateway is suspect, rotate |
+| `'user-cancel-fast'` | User cancelled quickly — gateway is probably fine, **keep it** |
+| _(omitted)_ | Same as `'load-error'` |
+
+The `excludeHost` option lets callers skip a specific gateway during retry without affecting the active gateway state.
+
+### Bypassing arweave.net
+
+arweave.net reliability can be spotty — the gateway may be up but unable to serve specific files. The `bypassArweaveNet` flag on `resolveUrl()` removes arweave.net from the resolution chain entirely. The chain becomes: persisted → fallback gateways → Wayfinder. Useful when arweave.net is unreliable but you don't want to remove it from the static gateway list permanently.
+
+To monitor arweave.net availability independently, use `checkGateway()` directly.
+
+### Bypassing Wayfinder
+
+The `bypassWayfinder` flag skips AR.IO Wayfinder resolution entirely. The chain becomes: persisted → arweave.net → static fallback gateways. Useful when you want to stick to known static gateways and avoid the latency of Wayfinder's composite routing (ping checks, network provider calls).
+
+### Gateway Prefetching
+
+[`prefetchUrls(urls, options?)`](#prefetchurlsurls-options) resolves multiple Arweave URLs in parallel with configurable concurrency (default: 5). Useful for warming the cache at startup with known model URLs or playlist images.
+
+---
+
+## Public API
+
+### Exports
+
+Everything is exported from `'playlist-data-engine'`:
+
+| Export | Type | Description |
+|--------|------|-------------|
+| `arweaveGatewayManager` | `ArweaveGatewayManager` | Pre-configured singleton instance |
+| `ArweaveGatewayManager` | Class | Create custom instances |
+| `GatewayConfig` | Interface | `{ host, protocol, priority }` |
+| `ArweaveGatewayManagerConfig` | Interface | Constructor options |
+| `ResolveUrlOptions` | Interface | Options for `resolveUrl()` (signal, bypassArweaveNet, bypassWayfinder) |
+| `GatewayCache` | Interface | Cache entry shape |
+| `GatewayCheckResult` | Interface | Resolution result with gateway info |
+| `PrefetchOptions` | Interface | Prefetch concurrency/behavior |
+| `PrefetchResult` | Interface | Prefetch outcome summary |
+| `PrefetchResultEntry` | Interface | Per-URL prefetch result |
+| `CacheStats` | Interface | Cache statistics |
+| `GatewayHealthStats` | Interface | Per-gateway health data |
+| `HealthCheckResult` | Interface | Health check outcome |
+| `HealthCheckOptions` | Interface | Health check configuration |
+| `isArweaveUrl` | Function | Check if a URL is an Arweave transaction |
+| `parseArweaveUrl` | Function | Extract txId and pathSuffix |
+| `constructGatewayUrl` | Function | Build gateway URL from components |
+| `getAllGatewayUrls` | Function | Get all gateway URLs for a txId in priority order |
+| `DEFAULT_GATEWAYS` | `GatewayConfig[]` | Default gateway list |
+| `KNOWN_GATEWAY_HOSTS` | `string[]` | Known gateway hostnames |
+| `ArweaveUrlInfo` | Type | Return type of `parseArweaveUrl()` |
+
+### Constructor: `ArweaveGatewayManagerConfig`
+
+```typescript
+import { ArweaveGatewayManager } from 'playlist-data-engine';
+
+const manager = new ArweaveGatewayManager({
+    gateways: [
+        { host: 'arweave.net', protocol: 'https', priority: 1 },
+        { host: 'ardrive.net', protocol: 'https', priority: 2 },
+    ],
+    timeout: 5000,
+    cacheTTL: 86400000,
+    slowResponseThreshold: 8000,
+    maxSlowResponses: 3,
+    solanaRpcUrl: 'https://solana-rpc.publicnode.com',
+    wayfinderSortBy: 'weights.compositeWeight',
+    wayfinderPrimaryLimit: 50,
+    wayfinderFallbackLimit: 100,
+    wayfinderStrategy: 'composite-ping-random',
+});
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `gateways` | `GatewayConfig[]` | 3 defaults | Custom gateway list (deep-cloned, sorted by priority) |
+| `timeout` | `number` | `5000` | Per-gateway HEAD check timeout (ms) |
+| `cacheTTL` | `number` | `86400000` | Per-txId cache TTL (24 hours) |
+| `slowResponseThreshold` | `number` | `8000` | Threshold above which a fetch is "slow" (ms) |
+| `maxSlowResponses` | `number` | `3` | Consecutive slow fetches before proactive gateway rotation |
+| `solanaRpcUrl` | `string` | `https://solana-rpc.publicnode.com` | Solana RPC URL for `@ar.io/sdk` gateway registry reads. See [Solana RPC Dependency](#solana-rpc-dependency). |
+| `wayfinderSortBy` | `WayfinderSortBy` | `weights.compositeWeight` | How AR.IO ranks gateways before building the pool. See [Tuning the Gateway Pool](#tuning-the-gateway-pool). |
+| `wayfinderPrimaryLimit` | `number` | `50` | Primary (FastestPing) pool size. |
+| `wayfinderFallbackLimit` | `number` | `100` | Fallback (Random) pool size. |
+| `wayfinderStrategy` | `WayfinderStrategy` | `composite-ping-random` | Routing strategy preset. See [Wayfinder Strategy Presets](#wayfinder-strategy-presets). |
+
+### Methods
+
+#### `resolveUrl(url, options?)`
+
+Main entry point. Resolves an Arweave URL to a working gateway URL using the sequential fallback system (persisted → arweave.net → static fallbacks → Wayfinder with retries). Non-Arweave URLs pass through unchanged.
+
+**Options (`ResolveUrlOptions`):**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `signal` | `AbortSignal` | — | Cancel in-flight gateway checks |
+| `bypassArweaveNet` | `boolean` | `false` | Skip arweave.net in the resolution chain — go persisted → fallbacks → Wayfinder |
+| `bypassWayfinder` | `boolean` | `false` | Skip Wayfinder in the resolution chain — only use static gateways |
+
+```typescript
+// Default behavior — arweave.net is in the chain
+const workingUrl = await arweaveGatewayManager.resolveUrl(
+    'https://arweave.net/abc123.../model.json',
+    { signal: abortController.signal }
+);
+
+// Bypass arweave.net — skip straight to fallbacks when arweave.net is unreliable
+const workingUrl = await arweaveGatewayManager.resolveUrl(
+    'https://arweave.net/abc123.../model.json',
+    { bypassArweaveNet: true }
+);
+
+// Bypass Wayfinder — stick to static gateways only
+const workingUrl = await arweaveGatewayManager.resolveUrl(
+    'https://arweave.net/abc123.../model.json',
+    { bypassWayfinder: true }
+);
+```
+
+#### `resolveUrlSimple(url)`
+
+Synchronous fast path — returns a gateway URL instantly by reusing the current active gateway (or arweave.net if none is set), with no network checks. Use this on the hot path where latency matters. When real fetches fail, call [`reportGatewayFailure()`](#reportgatewayfailureurl-options) to trigger gateway rotation.
+
+```typescript
+const url = arweaveGatewayManager.resolveUrlSimple(
+    'https://arweave.net/abc123.../model.json'
+);
+// → 'https://ardrive.net/abc123.../model.json' (immediate, no network)
+```
+
+#### `reportGatewayFailure(url, options?)`
+
+Reports that a real data fetch failed on the gateway that was returned by `resolveUrl()`. Clears the active gateway, persists the clearing to localStorage, and resolves a new gateway.
+
+```typescript
+const start = performance.now();
+try {
+    const response = await fetch(workingUrl, { signal });
+    if (!response.ok) throw new Error(`HTTP ${response.status}`);
+} catch (err) {
+    const elapsed = performance.now() - start;
+    const reason = err.name === 'AbortError'
+        ? (elapsed < 5000 ? 'user-cancel-fast' : 'user-cancel-slow')
+        : 'load-error';
+    newUrl = await arweaveGatewayManager.reportGatewayFailure(workingUrl, {
+        signal: abortController.signal,
+        reason,
+    });
+}
+```
+
+#### `reportFetchSuccess(timingMs)`
+
+Feeds timing data back from a successful fetch. If the fetch was above `slowResponseThreshold`, the consecutive slow counter increments. If fast, the counter resets.
+
+```typescript
+const start = performance.now();
+const response = await fetch(workingUrl);
+arweaveGatewayManager.reportFetchSuccess(performance.now() - start);
+```
+
+#### `reportFetchTiming(host, timingMs, success)`
+
+Records real data-transfer timing for a specific gateway host. This feeds into the health tracking system and affects priority adjustment and health-aware filtering.
+
+```typescript
+arweaveGatewayManager.reportFetchTiming('ar-io.net', 1250, true);
+```
+
+#### `prefetchUrls(urls, options?)`
+
+Warms up the gateway cache by resolving multiple URLs in parallel.
+
+```typescript
+const result = await arweaveGatewayManager.prefetchUrls([
+    'https://arweave.net/abc.../model.json',
+    'https://arweave.net/def.../model.json',
+], {
+    concurrency: 5,
+    continueOnError: true,
+});
+// result.succeeded, result.failed, result.skipped, result.entries
+```
+
+#### `getCacheStats()` / `clearCache()` / `getCachedGateway(txId)` / `setCache(txId, gateway)`
+
+Cache management. `getCacheStats()` returns size, hit/miss counts, and cached txIds.
+
+#### `getGatewayHealth(host)` / `getAllGatewayHealth()`
+
+Returns health statistics for one or all gateways based on recorded HEAD-check and fetch timing data.
+
+#### `adjustGatewayPriorities(options?)` / `resetGatewayPriorities()`
+
+Reorders gateways by health data (healthy first, then by speed) or resets to original configured priorities.
+
+#### `runHealthCheck(options?)`
+
+Runs a parallel HEAD check against all static gateways using a known transaction ID. Optionally adjusts gateway priorities based on results.
+
+#### `checkGateway(txId, gateway, pathSuffix?, signal?)`
+
+Verifies a single gateway can serve a transaction. Returns `true`, `false`, or `'maybe'` when the check is inconclusive.
+
+---
+
+## Internal Consumers
+
+These engine components use the gateway manager automatically — no configuration needed:
+
+| Component | What It Resolves | How |
+|-----------|-----------------|-----|
+| `PlaylistParser` | Playlist image URLs (`image_url`, `image_thumb_url`) | `arweaveGatewayManager.resolveUrl()` when `resolveImageUrls: true` |
+| `MusicClassifier` | TensorFlow model URLs | `arweaveGatewayManager.resolveUrl` as default `resolveUrl` option |
+| `EssentiaPitchDetector` | CREPE model URLs | `arweaveGatewayManager.resolveUrl` as default `resolveUrl` option |
+| `PitchAnalyzer` | CREPE model URLs | `arweaveGatewayManager.resolveUrl` as default `resolveUrl` option |
+| `ColorExtractor` | Track artwork image URLs | `arweaveGatewayManager.resolveUrl` directly |
+
+Each component also accepts an optional `resolveUrl` callback override if a consumer wants to provide custom resolution logic.
+
+---
+
+## Resolving Mix and Audio URIs
+
+The parser resolves image URLs only, and only when `resolveImageUrls: true` (see `PlaylistParser` above). Audio and mix URIs pass through parsing exactly as written — `ar://` identifiers, `ipfs://` paths, plain gateway URLs — and are not playable until something resolves them.
+
+Before playback, hand the URI to the gateway manager:
+
+```typescript
+import { arweaveGatewayManager } from 'playlist-data-engine';
+
+const url = await arweaveGatewayManager.resolveUrl(track.audio_url);
+
+// Play `url`. When the real fetch fails, rotate gateways and get a new URL:
+const retryUrl = await arweaveGatewayManager.reportGatewayFailure(url);
+```
+
+Mix URIs have one extra wrinkle: a mix can point at a metadata JSON file (`mime_type: 'application/json'`, or a `.json` path) rather than at audio, with the actual song named in that file's audio fields. [`resolveMixUrl()`](./PLAYLIST_PARSING.md#choosing-a-mix) wraps the same gateway resolution for mixes and follows that indirection one level — see [PLAYLIST_PARSING.md — Choosing a Mix](./PLAYLIST_PARSING.md#choosing-a-mix).
+
+---
+
+## Caching Layers
+
+The manager uses three separate caching mechanisms:
+
+1. **Per-txId in-memory cache** — Maps a 43-character transaction ID to a working `GatewayConfig`. TTL is configurable (default 24 hours). Checked first on every `resolveUrl()` call.
+2. **Active gateway persistence (localStorage)** — The single active gateway is persisted under key `arweave_active_gateway` with a 2-hour TTL. On page reload, the manager restores this gateway and tries it first (before arweave.net).
+3. **Wayfinder internal caching** — The `NetworkGatewaysProvider` caches the AR.IO gateway list internally. The `FastestPingRoutingStrategy` caches ping results. These are managed by the wayfinder library itself.
+
+---
+
+## Default Static Gateways
+
+| Priority | Host | Protocol |
+|----------|------|----------|
+| 1 | `arweave.net` | https |
+| 2 | `ardrive.net` | https |
+| 3 | `turbo-gateway.com` | https |
+
+These are tried in priority order after the persisted gateway. The persisted gateway from localStorage is tried first (known-working from previous session), then arweave.net, then the remaining static gateways in parallel. Wayfinder is the last resort with up to 3 retries.
+
+---
+
+## Externalized Dependencies
+
+The gateway manager requires `@ar.io/wayfinder-core` and `@ar.io/sdk` for Wayfinder functionality. Both are listed as regular dependencies in `package.json` (`@ar.io/wayfinder-core` 2.0.1, `@ar.io/sdk` 4.0.3) and externalized in the [vite.config.ts](../../vite.config.ts) rollup config:
+
+```typescript
+// vite.config.ts:19-20
+external: [
+    // ...
+    '@ar.io/wayfinder-core',
+    '@ar.io/sdk',
+],
+```
+
+The engine degrades gracefully if these packages are not available — static gateway fallback continues to work.
+
+---
+
+## Architecture Decisions
+
+**Why Wayfinder isn't the primary path** — Wayfinder requires pinging multiple gateways (up to 10), which takes 1-3 seconds. The engine tries the persisted gateway and arweave.net first because they're instant (single HEAD check, or cached hit). Static fallbacks come next. Wayfinder is only consulted as a last resort when all faster options fail.
+
+**Why Wayfinder results are verified** — Wayfinder's `resolveUrl()` returns a gateway that should work, but it can't guarantee the specific transaction is available on that gateway. The engine runs its own HEAD check against the resolved gateway to confirm the file exists before returning it. This prevents false positives.
+
+**Why dynamic imports** — `@ar.io/wayfinder-core` depends on Node.js `crypto` APIs. In browser environments, bundling it directly would require polyfills. The dynamic `import()` means it's loaded separately at runtime — if it fails, the manager falls back to static gateways.
+
+**Why localStorage persistence** — Without persistence, every page load would start from scratch: try arweave.net, fail, try static fallbacks, try Wayfinder (slow), etc. By remembering the last working gateway, subsequent page loads skip directly to a known-good gateway as the first step.
+
+**Why sequential fallback, not parallel** — Each step in the pipeline has different cost. A cache hit is free, arweave.net is usually fast, and Wayfinder is slow (1-3s of pinging). Running all in parallel would waste bandwidth and hit rate limits on gateways that will never be needed. Sequential fallback stops at the first success.
+
+**Why real HEAD checks, not pings** — The manager verifies each gateway can serve the *specific transaction*, not just that the gateway is up. A gateway that is healthy but doesn't have the data is correctly identified and skipped.
+
+---
+
+## Source Files
+
+| File | Purpose |
+|------|---------|
+| [src/utils/arweaveGatewayManager.ts](../../src/utils/arweaveGatewayManager.ts) | Gateway manager class with Wayfinder integration |
+| [src/utils/arweaveUtils.ts](../../src/utils/arweaveUtils.ts) | `GatewayConfig` interface, URL parsing, `DEFAULT_GATEWAYS`, utility functions |
+| [src/utils/ipfsUtils.ts](../../src/utils/ipfsUtils.ts) | IPFS URL detection, extraction, and resolution |
+| [src/index.ts](../../src/index.ts) | Public API exports |
+| [vite.config.ts](../../vite.config.ts) | Externalizes `@ar.io/wayfinder-core` and `@ar.io/sdk` from bundle |
+| [tests/unit/arweaveGatewayManager.test.ts](../../tests/unit/arweaveGatewayManager.test.ts) | Unit tests with mocked Wayfinder |
+
+---
+
+## URL Utilities
+
+The companion [arweaveUtils.ts](../../src/utils/arweaveUtils.ts) provides lower-level URL handling:
+
+| Export | Description |
+|--------|-------------|
+| `isArweaveUrl(url)` | Returns `true` for `ar://` protocol URLs or URLs containing known gateway hosts |
+| `parseArweaveUrl(url)` | Extracts txId and path suffix from `ar://` and `https://` Arweave URLs |
+| `constructGatewayUrl(txId, gateway, pathSuffix?)` | Builds a gateway URL from components |
+| `getAllGatewayUrls(txId, gateways?, pathSuffix?)` | Returns all gateway URLs for a txId in priority order |
+| `DEFAULT_GATEWAYS` | The three default gateway configs |
+| `KNOWN_GATEWAY_HOSTS` | Array of known gateway hostnames |
+
+---
+
+## IPFS URL Utilities
+
+The engine can detect and resolve IPFS URLs across multiple formats. NFT platforms like Sound.xyz, Catalog, and Spinamp frequently reference content (audio, images, artwork) via IPFS.
+
+### Supported URL Formats
+
+| Format | Example |
+|--------|---------|
+| Native scheme (double ipfs) | `ipfs://ipfs/QmXxx/path` |
+| Native scheme | `ipfs://QmXxx/path` |
+| Gateway URL | `https://ipfs.io/ipfs/QmXxx/path` |
+| Subdomain | `https://QmXxx.ipfs.dweb.link/path` |
+
+### Functions
+
+```typescript
+import { isIPFS, extractIPFSPath, resolveIPFSLink } from 'playlist-data-engine';
+
+// Detection
+isIPFS('https://soundxyz.mypinata.cloud/ipfs/QmXxx/file.jpg'); // true
+isIPFS('https://example.com/image.png');                         // false
+
+// Extract CID + path
+extractIPFSPath('ipfs://QmXxx/file.jpg');                         // 'QmXxx/file.jpg'
+extractIPFSPath('https://QmXxx.ipfs.dweb.link/file.jpg');        // 'QmXxx/file.jpg'
+
+// Resolve to a specific gateway (string param, backward compatible)
+resolveIPFSLink('ipfs://QmXxx');                                              // 'https://ipfs.io/ipfs/QmXxx'
+resolveIPFSLink('https://soundxyz.mypinata.cloud/ipfs/QmXxx', 'dweb.link');  // 'https://dweb.link/ipfs/QmXxx'
+
+// Preserve the original gateway when it's a recognized IPFS host
+resolveIPFSLink('https://soundxyz.mypinata.cloud/ipfs/QmXxx', { preserveOriginalGateway: true });
+// => 'https://soundxyz.mypinata.cloud/ipfs/QmXxx'
+
+// Force swap even with preserveOriginalGateway: false (same as default)
+resolveIPFSLink('https://soundxyz.mypinata.cloud/ipfs/QmXxx', {
+    gatewayHost: 'dweb.link',
+    preserveOriginalGateway: false,
+});
+// => 'https://dweb.link/ipfs/QmXxx'
+```
+
+Non-IPFS URLs pass through `resolveIPFSLink` unchanged.
+
+### Known Gateway Hosts
+
+The `isIPFS` and `extractIPFSPath` functions recognize these gateway hosts (exact match or subdomain):
+
+`ipfs.io`, `cloudflare-ipfs.com`, `dweb.link`, `gateway.pinata.cloud`, `ipfs.infura.io`, `nftstorage.link`, plus platform-specific hosts (Sound.xyz, Catalog, Spinamp, Bonfire, Web3 Music Pipeline).
+
+For the full list, see `KNOWN_IPFS_GATEWAY_HOSTS` in [src/utils/ipfsUtils.ts](../../src/utils/ipfsUtils.ts).
+
+### Exports
+
+| Export | Type | Description |
+|--------|------|-------------|
+| `isIPFS` | Function | Check if a URL is an IPFS URL (native scheme, gateway, or subdomain) |
+| `extractIPFSPath` | Function | Extract CID + path from any recognized IPFS URL format |
+| `resolveIPFSLink` | Function | Resolve IPFS URL to specified gateway; accepts string gateway host or `ResolveIPFSOptions` with `preserveOriginalGateway` |
+| `ResolveIPFSOptions` | Interface | Options for `resolveIPFSLink`: `gatewayHost?` and `preserveOriginalGateway?` |
+| `KNOWN_IPFS_GATEWAY_HOSTS` | `readonly string[]` | Recognized IPFS gateway hostnames |
+| `DEFAULT_IPFS_GATEWAY` | `string` | Gateway host used by `resolveIPFSLink` when none specified |
+
+---
+
+## Known Shortcomings & Future Improvements
+
+These are issues and improvement opportunities identified in the current implementation. None are bugs — the system works correctly as-is — but each represents an area where the implementation could be improved in a future iteration.
+
+### Hardcoded Health Filtering Thresholds
+
+The health filtering thresholds `HEALTHY_FAILURE_THRESHOLD` (0.70) and `MIN_CHECKS_FOR_FILTER` (3) are magic numbers inside `tryFallbackGateways()` ([arweaveGatewayManager.ts:650-651](../../src/utils/arweaveGatewayManager.ts#L650-L651)). These are not exposed in the constructor config. Making them configurable would allow consumers to tune the trade-off between resilience and gateway pool size.
+
+### No Circuit Breaker Pattern
+
+There is no mechanism to fully disable a failing gateway for a cooldown period. The health-aware filtering helps avoid known-dead gateways, but it only activates after 3+ checks and doesn't implement a time-based cooldown. A proper circuit breaker (e.g., open for 60s after N failures) would prevent repeatedly hitting a gateway that's temporarily down.
+
+### Tight Coupling to Browser APIs
+
+`localStorage` is accessed directly in `persistGateway()` and `loadPersistedGateway()`. This prevents the manager from being used in non-browser environments (Node.js, workers) without modification. Dependency injection for the persistence layer would improve testability and portability.
+
+### Unbounded Health Data Per Gateway
+
+`MAX_HEALTH_RECORDS` caps at 50 records per gateway, but there is no bound on the number of unique gateways that can accumulate health data. If `reportFetchTiming()` is called for arbitrary hosts (e.g., Wayfinder-returned gateways), the `healthData` map grows without bound over a long session.