@emircansahin/ghostfetch 0.1.0

package/README.md

@@ -0,0 +1,221 @@
+# ghostfetch
+
+Resilient HTTP client for Node.js with CycleTLS, automatic proxy rotation, smart error classification, and per-site custom interceptors.
+
+Built for backend developers who need to fetch data from sites that aggressively block automated requests.
+
+## Features
+
+- **CycleTLS** — TLS fingerprint spoofing (bypasses Cloudflare and similar WAFs)
+- **Proxy rotation** — random proxy selection per request with automatic health check
+- **Smart error classification** — proxy vs server vs ambiguous errors
+- **Proxy banning** — auto-ban failing proxies with configurable TTL
+- **Custom interceptors** — per-site response handling (`retry`, `ban`, `skip`)
+- **Default status handling** — 429/503 auto-retry, 407 proxy ban
+- **Cloudflare detection** — JS challenge detection with descriptive errors
+- **Country-based proxy selection** — auto-resolved via ipinfo.io
+- **forceProxy mode** — wait for available proxy instead of proceeding without one
+- **Health check on init** — dead proxies are discarded before any request
+
+## Install
+
+```bash
+npm install ghostfetch
+# or
+pnpm add ghostfetch
+```
+
+## Quick Start
+
+```ts
+import { GhostFetch } from 'ghostfetch';
+
+const client = new GhostFetch({
+  proxies: [
+    'http://user:pass@host:8001',
+    'http://user:pass@host:8002',
+  ],
+  timeout: 30000,
+  retry: { delays: [5000, 15000, 30000] }, // 3 retries: wait 5s, 15s, 30s
+  ban: { maxFailures: 3, duration: 60 * 60 * 1000 },
+  // ban: false — disable proxy banning entirely
+});
+
+// Wait for health check to complete
+const health = await client.ready();
+console.log(health);
+// { total: 2, healthy: 2, dead: 0, countries: { US: 1, DE: 1 }, proxies: { ... } }
+
+// Make requests
+const res = await client.get('https://api.example.com/data');
+console.log(res.status, res.body);
+```
+
+## Custom Interceptors
+
+Interceptors let you define per-site response handling. You can add them at the instance level (applies to all matching requests) or at the request level (applies to that single request only).
+
+### Instance-level interceptor
+
+Matches requests by URL. First matching interceptor takes full ownership — default status handling (429, 503, etc.) is bypassed.
+
+```ts
+client.addInterceptor({
+  name: 'example-api',
+  match: (url) => url.includes('example.com'),
+  check: (res) => {
+    if (res.status === 401) return 'skip';              // don't retry auth errors
+    if (res.body.includes('rate limit')) return 'retry'; // retry with different proxy
+    if (res.body.includes('blocked')) return 'ban';      // ban this proxy + retry
+    return null;                                          // use default behavior
+  },
+});
+```
+
+### Request-level interceptor
+
+No `match` needed — it applies to this specific request. Takes priority over instance-level interceptors.
+
+```ts
+const res = await client.get('https://special-api.com/data', {
+  interceptor: {
+    check: (res) => {
+      if (res.status === 401) return 'skip';
+      if (res.status === 200 && res.body.includes('error')) return 'retry';
+      return null;
+    },
+  },
+});
+```
+
+### Interceptor priority
+
+1. **Request-level interceptor** — checked first, if it returns non-null action, it wins
+2. **Instance-level interceptors** — checked next, first `match` takes ownership
+3. **Default status handling** — only runs if no interceptor claimed the response
+
+### Actions
+
+| Action | Proxy effect | Retry | Default bypass |
+|--------|-------------|-------|----------------|
+| `'retry'` | not penalized | yes | yes |
+| `'ban'` | fail count +1 | yes | yes |
+| `'skip'` | not penalized | no, return response | yes |
+| `null` | — | — | no, defaults apply |
+
+## Country-Based Proxy Selection
+
+All proxies are automatically resolved via ipinfo.io on init. This gives you country-level control over which proxy handles which request — useful when certain APIs only accept traffic from specific regions.
+
+```ts
+// Only use a German proxy for this request
+const res = await client.get('https://eu-only-api.com/data', {
+  country: 'DE',
+});
+```
+
+## Force Proxy Mode
+
+By default, if all proxies are banned or unavailable, requests proceed without a proxy. Enable `forceProxy` to make the request wait until a proxy becomes available (ban expires or proxy list is refreshed). This is useful for cron jobs where requests without a proxy are pointless.
+
+**What happens when `forceProxy: true` and no proxy is available?**
+
+| Scenario | Behavior |
+|----------|----------|
+| Proxy list is empty (none configured) | Throws `NoProxyAvailableError` immediately — nothing to wait for |
+| Proxies exist but all banned | Waits until a ban expires or `onProxyRefresh` provides fresh proxies |
+
+You can set it as the instance default or override per-request:
+
+```ts
+// Instance default: all requests wait for proxy
+const client = new GhostFetch({
+  proxies: [...],
+  forceProxy: true,
+});
+
+// Override per-request: this specific endpoint works without proxy
+const publicData = await client.get('https://public-api.com/data', {
+  forceProxy: false,
+});
+
+// Or the other way: instance default is false, but this request needs a proxy
+const protectedData = await client.get('https://protected-api.com/data', {
+  forceProxy: true,
+});
+```
+
+## Disable Proxy Banning
+
+If your proxy provider handles rotation internally (e.g. BrightData, Oxylabs residential) or you have a small proxy pool you don't want to lose, you can disable banning entirely:
+
+```ts
+const client = new GhostFetch({
+  proxies: [...],
+  ban: false, // no proxy will ever be banned
+});
+```
+
+With `ban: false`, failed proxies are never penalized — they stay in rotation regardless of errors. Retry still works, just without proxy exclusion logic.
+
+## Proxy Refresh
+
+Provide an `onProxyRefresh` callback to fetch a fresh proxy list from your provider. When triggered, all existing bans are cleared and the new proxies go through health check before entering the pool.
+
+| Config | Behavior |
+|--------|----------|
+| `onProxyRefresh` only | No automatic refresh — call `client.refreshProxies()` manually |
+| `onProxyRefresh` + `proxyRefreshInterval` | Auto-refresh at the given interval |
+| Neither | No refresh capability — initial proxy list is used for the lifetime |
+
+```ts
+// Auto-refresh every hour
+const client = new GhostFetch({
+  proxies: [...],
+  proxyRefreshInterval: 60 * 60 * 1000,
+  onProxyRefresh: async () => {
+    return ['http://user:pass@newhost:8001', ...];
+  },
+});
+
+// Or manual-only: no interval, call when you need it
+const client2 = new GhostFetch({
+  proxies: [...],
+  onProxyRefresh: async () => fetchFromProvider(),
+});
+await client2.refreshProxies(); // triggers onProxyRefresh → health check → pool updated
+```
+
+## Error Handling
+
+ghostfetch throws specific error classes so you can handle each scenario precisely:
+
+- **`CloudflareJSChallengeError`** — the target site requires a browser-level JS challenge that CycleTLS can't solve. You'll need puppeteer-extra with stealth plugin for this.
+- **`NoProxyAvailableError`** — all proxies are banned and `forceProxy` is not enabled, or the proxy list is empty.
+- **`MaxRetriesExceededError`** — all retry attempts failed. Contains `.attempts` count and `.lastError` with the final error details.
+
+```ts
+import {
+  CloudflareJSChallengeError,
+  NoProxyAvailableError,
+  MaxRetriesExceededError,
+} from 'ghostfetch';
+
+try {
+  const res = await client.get('https://example.com');
+} catch (err) {
+  if (err instanceof CloudflareJSChallengeError) {
+    // Needs headless browser (puppeteer-extra with stealth plugin)
+  }
+  if (err instanceof NoProxyAvailableError) {
+    // All proxies banned or list empty
+  }
+  if (err instanceof MaxRetriesExceededError) {
+    console.log(err.attempts, err.lastError);
+  }
+}
+```
+
+## License
+
+MIT

package/dist/cjs/classifier.js

@@ -0,0 +1,118 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.classifyError = classifyError;
+exports.isCloudflareChallenge = isCloudflareChallenge;
+exports.checkInterceptors = checkInterceptors;
+exports.checkDefaultRetryStatus = checkDefaultRetryStatus;
+/** Error codes that are definitely proxy/network failures — request never reached the server. */
+const PROXY_ERROR_CODES = new Set([
+    'ECONNREFUSED',
+    'ENOTFOUND',
+    'EAI_AGAIN',
+    'EHOSTUNREACH',
+    'ENETUNREACH',
+    'EPIPE',
+]);
+/** Error codes that could be proxy OR server — we can't tell for sure. */
+const AMBIGUOUS_ERROR_CODES = new Set([
+    'ETIMEDOUT',
+    'ECONNRESET',
+    'ECONNABORTED',
+]);
+const PROXY_ERROR_KEYWORDS = [
+    'proxy',
+    'tunnel',
+    'connect econnrefused',
+];
+const AMBIGUOUS_ERROR_KEYWORDS = [
+    'socket hang up',
+    'timeout',
+];
+/** Cloudflare JS challenge detection patterns */
+const CF_CHALLENGE_PATTERNS = [
+    'cf-browser-verification',
+    'cf_chl_opt',
+    'jschl_vc',
+    'jschl_answer',
+    'Checking your browser',
+    'Just a moment...',
+    '_cf_chl_tk',
+];
+/**
+ * Default status codes that should trigger retry.
+ * - 'server': retry with different proxy, proxy is not penalized
+ * - 'proxy': retry with different proxy, proxy fail count incremented
+ */
+const DEFAULT_RETRY_STATUSES = {
+    429: 'server', // Rate limit — not proxy's fault, just retry with different IP
+    503: 'server', // Service unavailable — server overloaded
+    407: 'proxy', // Proxy authentication required — proxy is broken
+};
+/**
+ * Classify an error as proxy, server, or ambiguous.
+ *
+ * - proxy:     request definitely never reached the server (DNS fail, connection refused, etc.)
+ * - server:    an HTTP response was received — the proxy worked fine
+ * - ambiguous: could be either (timeout, connection reset) — proxy should NOT be penalized
+ */
+function classifyError(error) {
+    if (error && typeof error === 'object') {
+        const err = error;
+        // If there's an HTTP status code, the request reached the server → server error
+        if (err.status && typeof err.status === 'number') {
+            return 'server';
+        }
+        const code = (err.code || err.errno);
+        const message = (err.message || '').toLowerCase();
+        // Check definite proxy errors first
+        if (code && PROXY_ERROR_CODES.has(code)) {
+            return 'proxy';
+        }
+        if (PROXY_ERROR_KEYWORDS.some((kw) => message.includes(kw))) {
+            return 'proxy';
+        }
+        // Check ambiguous errors
+        if (code && AMBIGUOUS_ERROR_CODES.has(code)) {
+            return 'ambiguous';
+        }
+        if (AMBIGUOUS_ERROR_KEYWORDS.some((kw) => message.includes(kw))) {
+            return 'ambiguous';
+        }
+    }
+    // Default: treat unknown errors as server errors (keep proxies alive)
+    return 'server';
+}
+/**
+ * Check if a successful response is actually a Cloudflare JS challenge.
+ */
+function isCloudflareChallenge(response) {
+    if (response.status === 403 || response.status === 503) {
+        return CF_CHALLENGE_PATTERNS.some((pattern) => response.body.includes(pattern));
+    }
+    return false;
+}
+/**
+ * Run interceptors against a response.
+ *
+ * First interceptor whose `match` returns true takes ownership.
+ * Its `check` result determines the action. Default status handling
+ * is bypassed whenever an interceptor matches (even if check returns null).
+ */
+function checkInterceptors(url, response, interceptors) {
+    for (const interceptor of interceptors) {
+        if (!interceptor.match(url))
+            continue;
+        const action = interceptor.check(response);
+        return { matched: true, action, interceptor };
+    }
+    return { matched: false, action: null };
+}
+/**
+ * Check if a response status code should trigger a default retry.
+ * Only called when no interceptor matched the URL.
+ * Returns the error type if retry should happen, or null if response is fine.
+ */
+function checkDefaultRetryStatus(status) {
+    return DEFAULT_RETRY_STATUSES[status] ?? null;
+}
+//# sourceMappingURL=classifier.js.map

package/dist/cjs/classifier.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"classifier.js","sourceRoot":"","sources":["../../src/classifier.ts"],"names":[],"mappings":";;AA2DA,sCAiCC;AAKD,sDAKC;AAkBD,8CAaC;AAOD,0DAEC;AA5ID,iGAAiG;AACjG,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC;IAChC,cAAc;IACd,WAAW;IACX,WAAW;IACX,cAAc;IACd,aAAa;IACb,OAAO;CACR,CAAC,CAAC;AAEH,0EAA0E;AAC1E,MAAM,qBAAqB,GAAG,IAAI,GAAG,CAAC;IACpC,WAAW;IACX,YAAY;IACZ,cAAc;CACf,CAAC,CAAC;AAEH,MAAM,oBAAoB,GAAG;IAC3B,OAAO;IACP,QAAQ;IACR,sBAAsB;CACvB,CAAC;AAEF,MAAM,wBAAwB,GAAG;IAC/B,gBAAgB;IAChB,SAAS;CACV,CAAC;AAEF,iDAAiD;AACjD,MAAM,qBAAqB,GAAG;IAC5B,yBAAyB;IACzB,YAAY;IACZ,UAAU;IACV,cAAc;IACd,uBAAuB;IACvB,kBAAkB;IAClB,YAAY;CACb,CAAC;AAEF;;;;GAIG;AACH,MAAM,sBAAsB,GAA8B;IACxD,GAAG,EAAE,QAAQ,EAAE,+DAA+D;IAC9E,GAAG,EAAE,QAAQ,EAAE,0CAA0C;IACzD,GAAG,EAAE,OAAO,EAAG,kDAAkD;CAClE,CAAC;AAEF;;;;;;GAMG;AACH,SAAgB,aAAa,CAAC,KAAc;IAC1C,IAAI,KAAK,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QACvC,MAAM,GAAG,GAAG,KAAgC,CAAC;QAE7C,gFAAgF;QAChF,IAAI,GAAG,CAAC,MAAM,IAAI,OAAO,GAAG,CAAC,MAAM,KAAK,QAAQ,EAAE,CAAC;YACjD,OAAO,QAAQ,CAAC;QAClB,CAAC;QAED,MAAM,IAAI,GAAG,CAAC,GAAG,CAAC,IAAI,IAAI,GAAG,CAAC,KAAK,CAAuB,CAAC;QAC3D,MAAM,OAAO,GAAG,CAAE,GAAG,CAAC,OAAkB,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,CAAC;QAE9D,oCAAoC;QACpC,IAAI,IAAI,IAAI,iBAAiB,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YACxC,OAAO,OAAO,CAAC;QACjB,CAAC;QAED,IAAI,oBAAoB,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC;YAC5D,OAAO,OAAO,CAAC;QACjB,CAAC;QAED,yBAAyB;QACzB,IAAI,IAAI,IAAI,qBAAqB,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YAC5C,OAAO,WAAW,CAAC;QACrB,CAAC;QAED,IAAI,wBAAwB,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC;YAChE,OAAO,WAAW,CAAC;QACrB,CAAC;IACH,CAAC;IAED,sEAAsE;IACtE,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED;;GAEG;AACH,SAAgB,qBAAqB,CAAC,QAA4B;IAChE,IAAI,QAAQ,CAAC,MAAM,KAAK,GAAG,IAAI,QAAQ,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;QACvD,OAAO,qBAAqB,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC;IAClF,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAWD;;;;;;GAMG;AACH,SAAgB,iBAAiB,CAC/B,GAAW,EACX,QAA4B,EAC5B,YAA2B;IAE3B,KAAK,MAAM,WAAW,IAAI,YAAY,EAAE,CAAC;QACvC,IAAI,CAAC,WAAW,CAAC,KAAK,CAAC,GAAG,CAAC;YAAE,SAAS;QAEtC,MAAM,MAAM,GAAG,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;QAC3C,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,WAAW,EAAE,CAAC;IAChD,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;AAC1C,CAAC;AAED;;;;GAIG;AACH,SAAgB,uBAAuB,CAAC,MAAc;IACpD,OAAO,sBAAsB,CAAC,MAAM,CAAC,IAAI,IAAI,CAAC;AAChD,CAAC"}