@crawlora-org/sdk 1.2.0-sdk.13 → 1.2.0-sdk.15

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## v1.2.0-sdk.15
+
+- Added case-insensitive request header overrides across auth, user-agent, and
+  content headers.
+- Added strict response mode validation, response headers on SDK errors, and
+  `Retry-After` aware retry delays capped at 30 seconds.
+- Clarified timeout and external abort errors without retrying user-aborted
+  requests.
+
+## v1.2.0-sdk.14
+
+- Aligned the promoted SDK beta tag with the Python and Go SDKs.
+- Added explicit coverage for request-level header overrides.
+
 ## v1.2.0-sdk.13
 
 - Updated the published npmjs package homepage to `https://crawlora.net`.

package/README.md

@@ -1,102 +1,154 @@
 # Crawlora JavaScript SDK
 
-JavaScript and TypeScript SDK for the public Crawlora API. It works in Node.js
-18+ and modern runtimes with `fetch`.
+Official JavaScript and TypeScript client for the public Crawlora API. Use it
+to call Crawlora scraping, search, marketplace, media, maps, finance, and usage
+endpoints with a small `fetch`-based client and generated TypeScript types.
 
-Website: [crawlora.net](https://crawlora.net)
+- Runtime: Node.js 18+ or any modern runtime with `fetch`
+- Auth: `x-api-key`
+- Default API base URL: `https://api.crawlora.net/api/v1`
+- Reference: [operations](docs/operations.md) and [recipes](docs/recipes.md)
 
-## Get an API Key
+## Install
 
-Create or sign in to your Crawlora account at [crawlora.net](https://crawlora.net),
-then open the dashboard and create an API key. Set it as `CRAWLORA_API_KEY` in
-your environment before running the examples or using the SDK.
+Install the latest published npm package:
 
-## Install
+```sh
+npm install @crawlora-org/sdk
+```
+
+For reproducible builds, pin a published package version:
 
 ```sh
-npm install @crawlora-org/sdk@latest
+npm install @crawlora-org/sdk@VERSION
 ```
 
-For reproducible builds, pin the current package version:
+## API Key
+
+Create or sign in to your Crawlora account at [crawlora.net](https://crawlora.net),
+then create an API key in the dashboard.
 
 ```sh
-npm install @crawlora-org/sdk@1.2.0-sdk.13
+read -r CRAWLORA_API_KEY
+export CRAWLORA_API_KEY
 ```
 
-## Usage
+## First Request
 
 ```js
 import { CrawloraClient } from "@crawlora-org/sdk";
 
-const crawlora = new CrawloraClient({ apiKey: process.env.CRAWLORA_API_KEY });
-const result = await crawlora.bing.search({ q: "coffee shops", count: 10 });
-console.log(result);
+const crawlora = new CrawloraClient({
+  apiKey: process.env.CRAWLORA_API_KEY
+});
+
+const response = await crawlora.bing.search({
+  q: "coffee shops",
+  count: 10
+});
+
+console.log(response.data?.results?.[0]);
 ```
 
-Generated TypeScript declarations cover operation ids, endpoint groups,
-parameter objects, enum values, request options, and response aliases.
-Dynamic calls infer parameter and response types from literal operation ids:
+Endpoint groups are generated from the public API contract, so common calls are
+available as methods such as `crawlora.bing.search(...)`,
+`crawlora.youtube.transcript(...)`, and `crawlora.google.mapSearch(...)`.
+
+## Typed Dynamic Calls
+
+You can also call by operation id. Literal operation ids infer parameter and
+response types in TypeScript:
 
 ```ts
-const result = await crawlora.request("bing-search", { q: "coffee shops" });
-result.data?.results?.[0]?.title;
+const response = await crawlora.request("bing-search", {
+  q: "coffee shops",
+  count: 10
+});
+
+response.data?.results?.[0]?.title;
 ```
 
+Generated declarations include operation ids, endpoint groups, request
+parameters, enum values, response aliases, and request options.
+
 ## Configuration
 
 ```js
 const crawlora = new CrawloraClient({
   apiKey: process.env.CRAWLORA_API_KEY,
   baseUrl: "https://api.crawlora.net/api/v1",
+  timeout: 30_000,
   retries: 2,
   retryDelay: 250,
-  headers: { "x-client": "example" }
+  headers: {
+    "x-client": "my-app"
+  }
 });
 ```
 
-Per-request options support headers, timeout, abort signals, and response mode:
+Per-request options can override headers, timeout, abort signal, and response
+mode:
 
 ```js
-const text = await crawlora.youtube.transcript(
-  { id: "VIDEO_ID", format: "text" },
-  { responseType: "text", timeout: 10000 }
+const controller = new AbortController();
+
+const response = await crawlora.bing.search(
+  { q: "coffee shops" },
+  {
+    timeout: 10_000,
+    signal: controller.signal,
+    headers: { "x-request-id": "search-001" }
+  }
 );
 ```
 
-Failed API calls throw `CrawloraError` with `status`, optional API `code`,
-parsed `body`, and the original `response` when available.
+## Text Responses
 
-## Reference
+Most endpoints return JSON. Endpoints that support alternate text output, such
+as YouTube transcripts, can opt into text mode:
 
-- [Operation reference](docs/operations.md)
-- [Usage recipes](docs/recipes.md)
+```js
+const transcript = await crawlora.youtube.transcript(
+  { id: "VIDEO_ID", format: "text" },
+  { responseType: "text" }
+);
 
-## Examples
+console.log(transcript);
+```
 
-Runnable examples live under `examples/`:
+## Errors
 
-```sh
-CRAWLORA_API_KEY=... npm run example:bing-search
-CRAWLORA_API_KEY=... CRAWLORA_YOUTUBE_VIDEO_ID=... npm run example:youtube-transcript
+Failed API calls throw `CrawloraError`:
+
+```js
+import { CrawloraClient, CrawloraError } from "@crawlora-org/sdk";
+
+try {
+  await crawlora.bing.search({ q: "coffee shops" });
+} catch (error) {
+  if (error instanceof CrawloraError) {
+    console.error(error.status, error.code, error.body);
+  }
+  throw error;
+}
 ```
 
-Each example also accepts `CRAWLORA_BASE_URL` for staging or local API testing.
-The examples exit without making a request when the required live environment
-variables are not set. `npm run smoke:live` runs all live examples in sequence.
+The error includes `status`, optional API `code`, parsed `body`, the original
+`response` when available, and a parser or transport `cause` when relevant.
+
+## Examples
 
-## Versioning
+Runnable examples live under `examples/` and skip cleanly when required
+environment variables are missing:
 
-This SDK is published to npmjs and mirrored to GitHub Packages. The package
-`latest` dist-tag and the moving Git tag named `latest` track the current
-promoted beta. Explicit package versions and Git beta tags such as
-`1.2.0-sdk.13` / `v1.2.0-sdk.13` remain available for reproducible builds. Pin
-an explicit version in production applications and upgrade intentionally.
+```sh
+npm run example:bing-search
+npm run example:youtube-transcript
+```
 
-GitHub Packages installs require a project `.npmrc` that maps the
-`@crawlora-org` scope to `https://npm.pkg.github.com` and a GitHub token that
-can read packages.
+Set `CRAWLORA_BASE_URL` to point examples at a staging or local API.
 
-## Package Name
+## Package Notes
 
 The npm package name is `@crawlora-org/sdk`:
 
@@ -104,11 +156,10 @@ The npm package name is `@crawlora-org/sdk`:
 import { CrawloraClient } from "@crawlora-org/sdk";
 ```
 
-The package names `crawlora` and `@crawlora/sdk` are not used by this repository
-because they already exist on npm and point to a different package source.
-
-## Optional Live Smoke Test
+The package names `crawlora` and `@crawlora/sdk` are not used by this
+repository because they already exist on npm and point to different package
+sources.
 
-Default tests use mock `fetch` implementations. The programs under `examples/`
-can be used as optional live smoke tests when `CRAWLORA_API_KEY` is available.
-Live calls are not part of default CI.
+This SDK is published to npmjs and may also be mirrored to GitHub Packages. Pin
+an explicit published version for production applications and upgrade
+intentionally.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@crawlora-org/sdk",
-  "version": "1.2.0-sdk.13",
+  "version": "1.2.0-sdk.15",
   "description": "JavaScript and TypeScript SDK for the public Crawlora API.",
   "keywords": [
     "crawlora",

package/src/client.js

@@ -1,18 +1,20 @@
 import { operations, groups } from "./operations.js";
 
 const DEFAULT_BASE_URL = "https://api.crawlora.net/api/v1";
-export const VERSION = "1.2.0-sdk.13";
+export const VERSION = "1.2.0-sdk.15";
 const DEFAULT_USER_AGENT = `crawlora-js-sdk/${VERSION}`;
 
 export class CrawloraError extends Error {
-  constructor(message, { status = 0, code, body, response, cause } = {}) {
+  constructor(message, { status = 0, code, body, headers = {}, response, cause, retryable } = {}) {
     super(message);
     this.name = "CrawloraError";
     this.status = status;
     this.code = code;
     this.body = body;
+    this.headers = headers;
     this.response = response;
     this.cause = cause;
+    this.retryable = retryable;
   }
 }
 
@@ -52,34 +54,39 @@ export class CrawloraClient {
     }
     params = params ?? {};
     options = options ?? {};
+    const responseType = validateResponseType(options.responseType || "auto");
 
     let attempt = 0;
     for (;;) {
       try {
-        return await this.#send(operation, params, options);
+        return await this.#send(operation, params, options, responseType);
       } catch (error) {
-        if (!(error instanceof CrawloraError) || attempt >= this.retries || !shouldRetry(error.status)) {
+        if (!(error instanceof CrawloraError) || error.retryable === false || attempt >= this.retries || !shouldRetry(error.status)) {
           throw error;
         }
         attempt++;
-        await sleep(retryDelay(this.retryDelay, attempt), options.signal);
+        await sleep(retryDelayForError(error, this.retryDelay, attempt), options.signal);
       }
     }
   }
 
-  async #send(operation, params, options) {
+  async #send(operation, params, options, responseType) {
     const { url, body, bodyHeaders } = buildRequest(operation, this.baseUrl, params);
-    const headers = {
-      ...this.headers,
-      ...authHeaders(operation.security, this.apiKey, this.jwtToken),
-      ...userAgentHeader(this.userAgent),
-      ...bodyHeaders,
-      ...(options.headers || {})
-    };
+    const headers = mergeHeaders(
+      this.headers,
+      authHeaders(operation.security, this.apiKey, this.jwtToken),
+      userAgentHeader(this.userAgent),
+      bodyHeaders,
+      options.headers
+    );
 
     const controller = new AbortController();
     const timeoutMs = options.timeout ?? this.timeout;
-    const timeout = timeoutMs > 0 ? setTimeout(() => controller.abort(), timeoutMs) : undefined;
+    let timedOut = false;
+    const timeout = timeoutMs > 0 ? setTimeout(() => {
+      timedOut = true;
+      controller.abort();
+    }, timeoutMs) : undefined;
     const signal = composeSignal(controller, options.signal);
     let response;
     try {
@@ -90,12 +97,19 @@ export class CrawloraClient {
         signal
       });
     } catch (error) {
+      if (options.signal?.aborted) {
+        throw new CrawloraError("Crawlora request aborted", { cause: error, retryable: false });
+      }
+      if (timedOut) {
+        throw new CrawloraError("Crawlora request timed out", { cause: error, retryable: false });
+      }
       throw new CrawloraError("Crawlora transport error", { cause: error });
     } finally {
       if (timeout) clearTimeout(timeout);
     }
 
-    const parsed = await parseResponse(response, options.responseType || "auto");
+    const headersObject = responseHeaders(response);
+    const parsed = await parseResponse(response, responseType, headersObject);
     if (!response.ok) {
       const code = parsed && typeof parsed === "object" ? parsed.code : undefined;
       const message = parsed && typeof parsed === "object" && parsed.msg ? parsed.msg : response.statusText;
@@ -103,6 +117,7 @@ export class CrawloraClient {
         status: response.status,
         code,
         body: parsed,
+        headers: headersObject,
         response
       });
     }
@@ -203,6 +218,22 @@ function userAgentHeader(userAgent) {
   return { "user-agent": userAgent };
 }
 
+function mergeHeaders(...sources) {
+  const headers = {};
+  const names = new Map();
+  for (const source of sources) {
+    for (const [name, value] of Object.entries(source || {})) {
+      if (value === undefined || value === null) continue;
+      const lower = name.toLowerCase();
+      const existing = names.get(lower);
+      if (existing && existing !== name) delete headers[existing];
+      headers[name] = String(value);
+      names.set(lower, name);
+    }
+  }
+  return headers;
+}
+
 function composeSignal(controller, signal) {
   if (!signal) return controller.signal;
   if (signal.aborted) controller.abort();
@@ -210,7 +241,14 @@ function composeSignal(controller, signal) {
   return controller.signal;
 }
 
-async function parseResponse(response, responseType) {
+function validateResponseType(responseType) {
+  if (responseType === "auto" || responseType === "json" || responseType === "text") {
+    return responseType;
+  }
+  throw new TypeError("Invalid responseType: expected one of auto, json, text");
+}
+
+async function parseResponse(response, responseType, headers) {
   if (responseType === "text") return response.text();
   const contentType = response.headers.get("content-type") || "";
   if (responseType === "json" || contentType.toLowerCase().includes("application/json")) {
@@ -221,6 +259,7 @@ async function parseResponse(response, responseType) {
       throw new CrawloraError("Crawlora JSON parse error", {
         status: response.status,
         body: text,
+        headers,
         response,
         cause: error
       });
@@ -240,6 +279,36 @@ function retryDelay(baseDelay, attempt) {
   return delay + jitter;
 }
 
+function retryDelayForError(error, baseDelay, attempt) {
+  return parseRetryAfter(error.headers) ?? retryDelay(baseDelay, attempt);
+}
+
+function parseRetryAfter(headers) {
+  const value = headerValue(headers, "retry-after");
+  if (!value) return undefined;
+  const seconds = Number(value);
+  if (Number.isFinite(seconds) && seconds > 0) {
+    return Math.min(seconds * 1000, 30000);
+  }
+  const date = Date.parse(value);
+  if (Number.isFinite(date)) {
+    const delay = date - Date.now();
+    if (delay > 0) return Math.min(delay, 30000);
+  }
+  return undefined;
+}
+
+function headerValue(headers, name) {
+  for (const [key, value] of Object.entries(headers || {})) {
+    if (key.toLowerCase() === name.toLowerCase()) return value;
+  }
+  return "";
+}
+
+function responseHeaders(response) {
+  return Object.fromEntries(response.headers.entries());
+}
+
 function normalizeNonNegativeInteger(value) {
   const number = Number(value);
   if (!Number.isFinite(number) || number <= 0) return 0;
@@ -256,13 +325,13 @@ function sleep(ms, signal) {
   if (!ms || ms <= 0) return Promise.resolve();
   return new Promise((resolve, reject) => {
     if (signal?.aborted) {
-      reject(new CrawloraError("Crawlora request aborted", { cause: signal.reason }));
+      reject(new CrawloraError("Crawlora request aborted", { cause: signal.reason, retryable: false }));
       return;
     }
     const timer = setTimeout(resolve, ms);
     signal?.addEventListener("abort", () => {
       clearTimeout(timer);
-      reject(new CrawloraError("Crawlora request aborted", { cause: signal.reason }));
+      reject(new CrawloraError("Crawlora request aborted", { cause: signal.reason, retryable: false }));
     }, { once: true });
   });
 }

package/src/index.d.ts

@@ -49,8 +49,10 @@ export class CrawloraError extends Error {
   status: number;
   code?: number;
   body: unknown;
+  headers: Record<string, string>;
   response?: Response;
   cause?: unknown;
+  retryable?: boolean;
 }
 
 export class CrawloraClient {