@datafuel/sdk 0.2.0 → 0.4.0

package/README.md

@@ -26,6 +26,8 @@ const markdown = await df.markdown("https://example.com");
 | A Google search              | `search`                                                | yes           |
 | Many prompts or searches     | `runAskJob` / `runSearchJob`                            | yes           |
 | Earlier jobs, tasks, usage   | `listJobs` / `listTasks` / `analytics` / `transactions` | yes           |
+| What a request may contain   | `jsInstructions` / `aiProviders` / `proxyLocations`     | yes           |
+| The wall in front of a URL   | `checkProtection`                                       | yes           |
 
 Start with plain `scrape`. Turn on `jsRendering` only when the page comes back empty: it is slower and costs five times the credits on a Basic proxy. `map` a section before you `crawl` it, it costs one credit and tells you how big it is.
 
@@ -63,7 +65,22 @@ try {
 
 `res.text` returns html or markdown, `res.data` structured output, `res.image` screenshot bytes.
 
-Page options, shared by `scrape`, jobs and crawls: `format` (`html`, `markdown`, `json`, `png`, `jpeg`), `jsRendering`, `waitFor`, `waitForTimeoutMs`, `jsInstructions` (an object keyed by action, e.g. `{ click: "#more" }`; `df.jsInstructions()` lists the actions), `blockResource`, `mainContentOnly`, `includeImages`, `extract`, `extractRegex`, `template`, `method`, `body`, `contentType`, `headers`, `headerOrder`, `cookies`, `userAgent`, `userAgentType`, `ai`.
+Page options, shared by `scrape`, jobs and crawls: `format` (`html`, `markdown`, `json`, `png`, `jpeg`), `jsRendering`, `waitFor`, `waitForTimeoutMs`, `jsInstructions`, `blockResource` (one resource type or an array, e.g. `["Image", "Font"]`), `mainContentOnly`, `includeImages`, `extract`, `extractRegex`, `template`, `method`, `body`, `contentType`, `headers`, `headerOrder`, `cookies`, `userAgent`, `userAgentType`, `ai`.
+
+`jsInstructions` is an array of single-action objects. They run in the order you list them and an action can repeat:
+
+```ts
+await df.scrape(url, {
+  jsRendering: true,
+  jsInstructions: [
+    { fill: ["input[name=q]", "laptops"] },
+    { click: "button[type=submit]" },
+    { wait_ms: 1000 },
+  ],
+});
+```
+
+The older form, one object keyed by action (`{ click: "#more" }`), still works, but its order is not guaranteed and an action cannot repeat. `df.jsInstructions()` lists the actions.
 
 `proxy` takes `type`, `country`, `city`, `state`, `asn`, and a sticky `sessionId` with `ttl` (seconds) on `scrape`, `map`, URL jobs and crawls. `df.proxyLocations()` and `df.proxyAsns(country)` list what a proxy type can exit from.
 
@@ -76,15 +93,14 @@ const res = await df.scrape(url, {
   ai: {
     prompt: "extract the product name and its price",
     format: { name: "string", price: "number" },
-    provider: "openai", // openai, anthropic, google
-    model: "gpt-4o-mini",
+    provider: "openai", // required; df.aiProviders() lists the providers and their models
     apiKey: process.env.OPENAI_API_KEY,
   },
 });
 res.data; // { name: "...", price: ... }
 ```
 
-Works on `scrape` and on URL jobs. Crawls reject it; the SDK says so before sending. `provider`, `model` and `apiKey` must be given together — a half-filled key would be spent on a scrape whose AI step then fails.
+Works on `scrape` and on URL jobs. Crawls reject it; the SDK says so before sending. `provider` is required. `model` is optional: leave it out for the provider's default, or pick one of the models `df.aiProviders()` lists; any other value is rejected with `InvalidAttributes`.
 
 ## Crawl
 
@@ -141,7 +157,7 @@ const serp = await df.search("best crm", { country: "us", language: "en", page:
 console.log(serp.data); // parsed results page; format "html" or "markdown" for the raw page
 ```
 
-`search` also takes `location` (or `uule`, or `lat`/`lon` with `radius`), `googleDomain`, `tbs`, `safe`, `cr`, `lr`, `nfpr`, `filter` and `proxyCountry`.
+`search` also takes `location` (or `uule`, or `lat`/`lon` with `radius`), `googleDomain`, `tbs`, `safe`, `cr`, `lr`, `nfpr` and `filter`. Searches leave through DataFuel's own pool: pick the market with `country` and `language`; `proxyCountry` is accepted but not used yet.
 
 ## Map
 
@@ -165,6 +181,23 @@ const history = await df.transactions({ operation: "refund", limit: 50 });
 
 `listJobs` and `listTasks` run newest first; pass `nextCursor` back as `cursor` until it is absent. Task items carry no result: call `getTask(id)` for it. Dates are `YYYY-MM-DD` in UTC (a `Date` is sent as its UTC day) and `endDate` is inclusive. `transactions` pages with `page` and `limit`, and its `sums` total each operation over the whole range. In `analytics`, `status_code` 0 means the target never answered (timeout, DNS).
 
+```ts
+const credits = await df.balance();
+const { plan_balance, payg_balance } = await df.balanceSplit();
+```
+
+`balance` is what you can spend. It is made of plan credits and pay-as-you-go credits. Plan credits are spent first; unused ones roll over when the plan renews and expire if it is not renewed. Pay-as-you-go credits come from one-time credit packs (a `purchase` transaction), are spent after plan credits and never expire. Each transaction's `plan_amount` is the part of `amount` that moved plan credits.
+
+## Config and health
+
+```ts
+await df.capabilities(); // which task types and LLM engines are on
+await df.jsInstructions(); // the browser actions jsInstructions accepts
+await df.aiProviders(); // the LLM providers and models ai accepts
+await df.checkProtection(["https://shop.example.com/"]); // anti-bot vendor per URL, nothing scraped
+const health = await df.health({ deep: true }); // health.ok is false while a dependency is down
+```
+
 ## Errors
 
 ```ts
@@ -184,7 +217,7 @@ try {
 ```
 
 - `NoApiKey`: no key was passed and `DATAFUEL_API_KEY` is empty. Thrown before any request.
-- `APIError`: the API refused the request. `.code` holds the API's error code (typed as `ErrorCode`). Subclasses: `Unauthorized`, `Forbidden`, `InsufficientCredits`, `RateLimited`, `NotFound`, `InvalidAttributes`, `IdempotencyKeyReused`, `JobNotCancellable`.
+- `APIError`: the API refused the request. `.code` holds the API's error code (typed as `ErrorCode`). Subclasses: `Unauthorized`, `Forbidden`, `InsufficientCredits`, `RateLimited`, `NotFound`, `InvalidAttributes`, `IdempotencyKeyReused`, `JobNotCancellable`, `AlreadyExists` (a create collided with an existing task or job; nothing was charged, send it again).
 - `ModuleUnavailable`, `EngineUnavailable`: an operator switched a task type or LLM engine off, e.g. during a provider outage. The reason is in the message, nothing is charged, and the SDK does not retry. `df.capabilities()` lists what is on.
 - `TaskFailed`, and `Blocked` when the target refused: the API accepted the task but the page could not be scraped. The error carries `.result`, so the envelope is still readable. Failed tasks are refunded.
 - `WaitTimeout`: a wait ran out of time. `.id` picks the work back up.
@@ -217,7 +250,7 @@ Keep `timeoutMs` generous: `scrape` waits until the page is ready, which can tak
 
 If you pass your own `fetch`, leave redirects off. fetch keeps custom headers across a redirect, so a redirect to another host would carry your `X-API-Key` to it. This SDK sends `redirect: "manual"`.
 
-Full API reference: https://scraping-api.datafuel.ai/docs
+Guides and the full API reference: https://docs.datafuel.ai
 
 ## MCP
 

package/dist/index.cjs

@@ -21,6 +21,7 @@ var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: tru
 var index_exports = {};
 __export(index_exports, {
   APIError: () => APIError,
+  AlreadyExists: () => AlreadyExists,
   Blocked: () => Blocked,
   Capabilities: () => Capabilities,
   CrawlPage: () => CrawlPage,
@@ -87,6 +88,8 @@ var IdempotencyKeyReused = class extends APIError {
 };
 var JobNotCancellable = class extends APIError {
 };
+var AlreadyExists = class extends APIError {
+};
 var Unavailable = class extends APIError {
 };
 var ModuleUnavailable = class extends Unavailable {
@@ -124,14 +127,15 @@ var BY_CODE = {
   IDEMPOTENCY_KEY_REUSED: IdempotencyKeyReused,
   INVALID_API_KEY: Unauthorized,
   FORBIDDEN: Forbidden,
-  JOB_NOT_CANCELLABLE: JobNotCancellable
+  JOB_NOT_CANCELLABLE: JobNotCancellable,
+  TASK_ALREADY_EXISTS: AlreadyExists,
+  JOB_ALREADY_EXISTS: AlreadyExists
 };
 var BY_STATUS = {
   401: Unauthorized,
   402: InsufficientCredits,
   403: Forbidden,
   404: NotFound,
-  409: JobNotCancellable,
   422: IdempotencyKeyReused,
   429: RateLimited,
   503: Unavailable
@@ -152,19 +156,20 @@ function apiError(status, body, retryAfter = 0) {
 
 // src/core.ts
 var DEFAULT_BASE_URL = "https://scraping-api.datafuel.ai/api/v1";
-var VERSION = "0.2.0";
+var VERSION = "0.4.0";
 var DEFAULT_TIMEOUT_MS = 18e4;
 var STILL_PROCESSING_DELAY_MS = 2e3;
 var STILL_PROCESSING = "TASK_STILL_PROCESSING";
 var MAX_IDEMPOTENCY_KEY = 255;
 var Request = class {
-  constructor(method, path, params, body, idempotencyKey, auth = true) {
+  constructor(method, path, params, body, idempotencyKey, auth = true, degradedOk = false) {
     this.method = method;
     this.path = path;
     this.params = params;
     this.body = body;
     this.idempotencyKey = idempotencyKey;
     this.auth = auth;
+    this.degradedOk = degradedOk;
   }
   method;
   path;
@@ -172,6 +177,7 @@ var Request = class {
   body;
   idempotencyKey;
   auth;
+  degradedOk;
   /** GETs are safe by nature, writes because they carry an idempotency key. */
   get retryable() {
     return this.method === "GET" || this.idempotencyKey !== void 0;
@@ -219,15 +225,8 @@ function aiAttributes(ai) {
   return out;
 }
 function validateAI(ai) {
-  const given = {
-    provider: ai.provider !== void 0,
-    model: ai.model !== void 0,
-    apiKey: ai.apiKey !== void 0
-  };
-  const present = Object.values(given).filter(Boolean).length;
-  if (present > 0 && present < 3) {
-    const missing = Object.entries(given).filter(([, ok]) => !ok).map(([name]) => name).join(", ");
-    throw new TypeError(`ai needs provider, model and apiKey together; missing: ${missing}`);
+  if (!ai.provider) {
+    throw new TypeError("ai needs a provider; aiProviders() lists the ones the API supports");
   }
 }
 function scrapeAttributes(options = {}) {
@@ -237,7 +236,9 @@ function scrapeAttributes(options = {}) {
   if (options.waitFor) attrs.wait_for_selector = options.waitFor;
   if (options.waitForTimeoutMs) attrs.wait_for_selector_timeout_ms = options.waitForTimeoutMs;
   if (options.jsInstructions) attrs.js_instructions = options.jsInstructions;
-  if (options.blockResource) attrs.block_resource = options.blockResource;
+  if (options.blockResource?.length) {
+    attrs.block_resource = typeof options.blockResource === "string" ? options.blockResource : [...options.blockResource];
+  }
   if (options.mainContentOnly) attrs.main_content_only = true;
   if (options.includeImages !== void 0) attrs.include_images = options.includeImages;
   if (options.extract) attrs.extract_selector = selector(options.extract);
@@ -419,6 +420,17 @@ function crawlResultsRequest(crawlId, cursor, limit) {
     Object.keys(params).length > 0 ? params : void 0
   );
 }
+function healthRequest(deep) {
+  return new Request(
+    "GET",
+    "/healthz",
+    deep ? { deep: "1" } : void 0,
+    void 0,
+    void 0,
+    false,
+    true
+  );
+}
 function day(value) {
   return typeof value === "string" ? value : value.toISOString().slice(0, 10);
 }
@@ -651,7 +663,7 @@ var DataFuel = class {
         const signal = this.signalFor(opts);
         const send = this.fetchImpl;
         const response = await send(url, signal ? { ...init, signal } : init);
-        return await parse(response);
+        return await parse(response, request.degradedOk);
       } catch (caught) {
         if (isAbort(caught)) {
           throw new TransportError("the request was aborted or timed out", { cause: caught });
@@ -1013,6 +1025,37 @@ var DataFuel = class {
     const body = record(await this.send(request, options));
     return Array.isArray(body.instructions) ? body.instructions : [];
   }
+  /** The LLM providers and models `ai` accepts. Needs no key. */
+  async aiProviders(options = {}) {
+    const request = new Request(
+      "GET",
+      "/config/ai-providers",
+      void 0,
+      void 0,
+      void 0,
+      false
+    );
+    const body = record(await this.send(request, options));
+    return Array.isArray(body.providers) ? body.providers : [];
+  }
+  /**
+   * Whether the API is up. `deep` also checks the dependencies it needs to
+   * serve scrapes. A degraded report is returned, not thrown: read `ok`.
+   * Needs no key.
+   */
+  async health(options = {}) {
+    const body = record(await this.send(healthRequest(options.deep ?? false), options));
+    return { ...body, ok: body.status === "ok" };
+  }
+  /**
+   * Which anti-bot protection sits in front of each URL. Nothing is scraped
+   * and nothing is charged; invalid URLs are skipped.
+   */
+  async checkProtection(urls, options = {}) {
+    return list(
+      await this.send(new Request("POST", "/filter/check", void 0, [...urls]), options)
+    );
+  }
   /** Countries, regions and cities a proxy type can exit from. */
   async proxyLocations(options = {}) {
     const params = options.proxyType ? { proxy_type: options.proxyType } : void 0;
@@ -1028,13 +1071,22 @@ var DataFuel = class {
       await this.send(new Request("GET", "/config/proxy/asn", params), options)
     );
   }
-  /** Remaining credits. */
+  /** Remaining credits: plan and pay-as-you-go together. */
   async balance(options = {}) {
     const body = await this.send(new Request("GET", "/users/@me/balance"), options);
     return intField(body, "balance");
   }
+  /** Remaining credits by pool: plan credits (spent first) and pay-as-you-go credits. */
+  async balanceSplit(options = {}) {
+    const body = await this.send(new Request("GET", "/users/@me/balance"), options);
+    return {
+      balance: intField(body, "balance"),
+      plan_balance: intField(body, "plan_balance"),
+      payg_balance: intField(body, "payg_balance")
+    };
+  }
   /**
-   * Credit movements, newest first: purchases, usage, refunds, expiry.
+   * Credit movements, newest first: plan assignments, credit pack purchases, usage, refunds, expiry.
    * `sums` totals each operation over the whole range, not just this page.
    */
   async transactions(options = {}) {
@@ -1075,7 +1127,7 @@ var DataFuel = class {
 function envApiKey() {
   return typeof process !== "undefined" ? process.env?.DATAFUEL_API_KEY : void 0;
 }
-async function parse(response) {
+async function parse(response, degradedOk = false) {
   const text = await response.text();
   let body;
   if (text.length > 0) {
@@ -1085,6 +1137,7 @@ async function parse(response) {
       body = text;
     }
   }
+  if (degradedOk && response.status === 503 && isHealthReport(body)) return body;
   if (!response.ok) {
     const header = response.headers.get("Retry-After");
     const retryAfter = header !== null && !Number.isNaN(Number(header)) ? Number(header) : 0;
@@ -1092,6 +1145,9 @@ async function parse(response) {
   }
   return body;
 }
+function isHealthReport(body) {
+  return body !== null && typeof body === "object" && "status" in body;
+}
 function isAbort(error) {
   return error instanceof Error && (error.name === "AbortError" || error.name === "TimeoutError");
 }
@@ -1149,6 +1205,7 @@ function intField(body, name) {
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
   APIError,
+  AlreadyExists,
   Blocked,
   Capabilities,
   CrawlPage,