minifetch-api 0.0.1 → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lauren Garcia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,26 +1,120 @@
-# Minifetch.com API
+# Minifetch API
+<div align=center>
+  <img src="https://minifetch.com/minifetch-dog-logo--whitebg.png" width="70" />
+</div>
 
-Works with x402 payments on Base Sepolia Testnet (only) so far. Testnet API endpoints are not exposed. Check back again later!
+**Fetch & extract data from web pages.** [Minifetch.com](https://minifetch.com) provides composable extraction APIs that humans and AI Agents can autonomously discover, orchestrate, and pay for— making web pages simple to access. We offer competitive prices.
 
-This client based on x402 payments [example client code](https://github.com/coinbase/x402/blob/main/examples/typescript/clients/fetch).
+**Full [API docs are here](https://minifetch.com/docs/api)**, including example response data & prices. For AI Agents, read the [LLMs.txt](https://minifetch.com/llms.txt).
+
+**Payments** work with [x402](https://www.x402.org/) USDC stablecoin micropayments on Coinbase's Base & Solana blockchain networks. Transaction fees are free. If you'd like to use a traditional credit card & API key instead, sign up for the waitlist and we'll let you know when it's ready: [forms.gle/rkMi7T23bHJc8XFw9](https://forms.gle/rkMi7T23bHJc8XFw9)
+
+**Bulk fetch and extract** for building datasets is coming soon, sign up for the waitlist and we'll let you know when it's ready: [forms.gle/rkMi7T23bHJc8XFw9](https://forms.gle/rkMi7T23bHJc8XFw9)
 
 ## Prerequisites
 
-- Node.js v20+ (install via [nvm](https://github.com/nvm-sh/nvm))
-- pnpm v10 (install via [pnpm.io/installation](https://pnpm.io/installation))
-- A valid Ethereum private key for making USDC payments on Base Sepolia Testnet (only) -- live production client coming soon!
+- Node.js v18+ & NPM
+- A valid Ethereum or Solana private key for making USDC payments.
 
-## Setup
+## Install
 
-1. Copy `.env-local` to `.env-base-testnet` and add your Ethereum private key:
-```bash
-cp .env-local .env-base-testnet
-```
+`npm install minifetch-api --save`
+
+## Quick Start
 
-2. Start the example client:
-```bash
-pnpm install
-pnpm dev:testnet:check
-pnpm dev:testnet
+```js
+import Minifetch from "minifetch-api";
+
+// Inititialize the client with your network choice & private key.
+//   - Network options: "base" or "solana"
+//   - Use private key from account that has a small amt of USDC
+const client = new Minifetch({
+  network: "base",
+  privateKey: process.env.BASE_PRIVATE_KEY,
+});
+
+try {
+  const url = "example.com";
+  const response = await client.checkAndExtractPreview(url);
+  // 200 "ok" responses will return the data and x402 payment info:
+  console.log("data: ", response.data);
+  console.log("payment info: ", response.payment);
+} catch (err) {
+  // No payment or charges for errors!
+  console.log(err);
+}
 ```
 
+### Client Methods Available
+
+After you initialize the client (above), you have the following methods available to use. The "checkAndExtract" API methods provided help you avoid paying for blocked URLs and return more granular info about why a URL may or may not return data. There is no charge for blocked upstream URLs.
+
+[Price list and example data for each method is here.](https://minifetch.com/docs/api#example-data)
+
+Requests exceeding rate limits return `503` errors. This is intentional — back off and retry, max 5-10 queries per second. We'll add bulk fetch and extract in the future. Sign up for the [waitlist here](https://forms.gle/rkMi7T23bHJc8XFw9).
+
+- `client.checkAndExtractUrlPreview(url)`
+  - Extracts a light, token-efficient preview of a URL: title, description, and image (only).
+
+- `client.checkAndExtractUrlContent(url, options)`
+  - Options: `includeMediaUrls: true`, defaults to false
+  - Extracts a clean, token-efficient content summary as markdown from a URL. Removes ads, nav, scripts. Much more efficient than raw HTML fetches for LLMs.
+
+- `client.checkAndExtractUrlLinks(url)`
+  - Extracts all links from a URL categorized by type (internal/external/anchor) with SEO metadata. Detects image links, nofollow attributes, and analyzes external domain distribution.
+
+- `client.checkAndExtractUrlMetadata(url, options)`
+  - Options: `includeResponseBody: true`, defaults to false
+  - Extracts rich structured metadata from a URL: title, description, og/twitter tags, json-ld, images, headings, response headers, and more. Perfect for SEO research or metadata indexing & analysis. Largest response size of the client methods available.
+
+For maximum control, you can also use the following methods directly:
+
+- `client.preflightCheck(url)`
+  - Standalone call to check if a URL is allowed to be fetched according to the website's robots.txt file. Use this free endpoint before using our other paid endpoints to avoid spending extra on un-fetchable URLs
+- `client.extractUrlPreview(url)`
+- `client.extractUrlContent(url, options)`
+  - Options: `includeMediaUrls: true`, defaults to false
+- `client.extractUrlLinks(url)`
+- `client.extractUrlMetadata(url, options)`
+  - Options: `includeResponseBody: true`, defaults to false
+
+### Error Types
+When you wrap the functions above in a try/catch, here are the errors you may encounter:
+
+- **"RobotsBlockedError: URL is blocked by robots.txt"**
+  - URL is explicitly blocked by the website's robots.txt will error like this.
+- **"Network Error: 402 Payment Required"**
+  - Check your wallet -- likely you ran out of USDC to pay!
+- **"Network Error: 502 Bad Gateway"**
+  - URLs that pass the robots.txt check but are blocked anyway via 403 or other tactics may error like this.
+- **"503 Service Temporarily Unavailable"**
+  - Likely encountering Minifetch rate-limiting or upstream timeout errors on the target URL. You may try again but limit requests to 5-10 at a time. Bulk fetches coming soon.
+- **"InvalidURLError: Invalid url ${url}"**
+  - The URL you passed in is malformed in some way, correct it and try again.
+
+### Service Limitations
+Minifetch only extracts publicly available metadata and content from pages accessible without authentication and javascript execution.
+
+What Minifetch does NOT do:
+
+- Ignore robots.txt directives
+- Access authenticated or logged-in content
+- Create accounts or log into user sessions
+- Perform transactional actions (checkout, bidding, purchasing, form submissions)
+- Bypass paywalls or access restricted content
+
+What Minifetch does NOT do *currently* but may offer in the future as an add-on:
+- Javascript execution
+
+## x402 Best Practices
+
+- LLMs & Agents should *never* have direct access to your private key. They *will* expose it!
+- Only keep a small amount of USDC in the account whose private key you use.
+- Keep that account separate from the rest of your onchain funds.
+- Pass your private key into the Minifetch API Client via an [environment variable](https://developer.vonage.com/en/blog/how-to-use-environment-variables-in-javascript-with-dotenv).
+
+---
+**License**
+
+MIT / Copyright (c) 2026 Lauren Garcia
+This package is an API client for Minifetch.com. The client code is open source, but use of the Minifetch API is subject to the [Minifetch.com Terms of Service](https://minifetch.com/terms-of-service).

package/dist/esm/client.js

@@ -0,0 +1,266 @@
+import { initConfig } from "./init.js";
+import { validateAndNormalizeUrl } from "./utils/validation.js";
+import { handlePayment } from "./utils/payment.js";
+import { InvalidUrlError, RobotsBlockedError, PaymentFailedError, ExtractionFailedError, NetworkError, } from "./types/errors.js";
+/**
+ * Main Minifetch API client
+ * Provides methods to check URLs and extract metadata/links/preview/content
+ */
+export class MinifetchClient {
+    config;
+    baseUrl;
+    /**
+     *
+     * @param config
+     */
+    constructor(config) {
+        this.config = initConfig(config);
+        this.baseUrl = this.config.apiBaseUrl;
+    }
+    /**
+     * Check if URL is allowed by robots.txt (free preflight check)
+     *
+     * @param url
+     * @throws {InvalidUrlError} if URL is invalid
+     * @throws {NetworkError} if request fails
+     */
+    async preflightUrlCheck(url) {
+        try {
+            // Validate and normalize URL
+            const normalizedUrl = validateAndNormalizeUrl(url);
+            // Build request URL (free endpoint, no payment)
+            const endpoint = `/api/v1/free/preflight/url-check?url=${encodeURIComponent(normalizedUrl)}`;
+            const requestUrl = `${this.baseUrl}${endpoint}`;
+            const response = await fetch(requestUrl);
+            if (!response.ok) {
+                throw new NetworkError(`Preflight check failed: ${response.status} ${response.statusText}`);
+            }
+            const data = (await response.json());
+            return data;
+        }
+        catch (error) {
+            if (error instanceof InvalidUrlError || error instanceof NetworkError) {
+                throw error;
+            }
+            throw new NetworkError(`Preflight check failed: ${error instanceof Error ? error.message : "Unknown error"}`);
+        }
+    }
+    /**
+     * Extract URL metadata (paid, requires x402 payment)
+     *
+     * @param url
+     * @param options
+     * @param options.includeResponseBody
+     * @throws {InvalidUrlError} if URL is invalid
+     * @throws {PaymentFailedError} if payment fails
+     * @throws {ExtractionFailedError} if extraction fails
+     */
+    async extractUrlMetadata(url, options) {
+        try {
+            // Validate and normalize URL
+            const normalizedUrl = validateAndNormalizeUrl(url);
+            // Build request URL with optional params
+            const params = new URLSearchParams({ url: normalizedUrl });
+            if (options?.includeResponseBody) {
+                params.set("includeResponseBody", "true");
+            }
+            // Build request URL - convert params to string
+            const endpoint = `/api/v1/x402/extract/url-metadata?${params.toString()}`;
+            const requestUrl = `${this.baseUrl}${endpoint}`;
+            // Make request with x402 payment handling
+            const { response, payment } = await handlePayment(requestUrl, this.config);
+            // Check if extraction succeeded
+            if (!response.ok) {
+                throw new ExtractionFailedError(normalizedUrl, `Metadata extraction failed: ${response.status} ${response.statusText}`);
+            }
+            const data = (await response.json());
+            return {
+                success: data.success,
+                results: data.results,
+                payment,
+            };
+        }
+        catch (error) {
+            if (error instanceof InvalidUrlError ||
+                error instanceof ExtractionFailedError ||
+                error instanceof PaymentFailedError ||
+                error instanceof NetworkError) {
+                throw error;
+            }
+            throw new ExtractionFailedError(url, `Metadata extraction failed: ${error instanceof Error ? error.message : "Unknown error"}`);
+        }
+    }
+    /**
+     * Extract URL links (paid, requires x402 payment)
+     *
+     * @param url
+     * @throws {InvalidUrlError} if URL is invalid
+     * @throws {PaymentFailedError} if payment fails
+     * @throws {ExtractionFailedError} if extraction fails
+     */
+    async extractUrlLinks(url) {
+        try {
+            const normalizedUrl = validateAndNormalizeUrl(url);
+            const endpoint = `/api/v1/x402/extract/url-links?url=${encodeURIComponent(normalizedUrl)}`;
+            const requestUrl = `${this.baseUrl}${endpoint}`;
+            // Make request with x402 payment handling
+            const { response, payment } = await handlePayment(requestUrl, this.config);
+            if (!response.ok) {
+                throw new ExtractionFailedError(normalizedUrl, `Links extraction failed: ${response.status} ${response.statusText}`);
+            }
+            const data = (await response.json());
+            return {
+                success: data.success,
+                results: data.results,
+                payment,
+            };
+        }
+        catch (error) {
+            if (error instanceof InvalidUrlError ||
+                error instanceof ExtractionFailedError ||
+                error instanceof PaymentFailedError ||
+                error instanceof NetworkError) {
+                throw error;
+            }
+            throw new ExtractionFailedError(url, `Links extraction failed: ${error instanceof Error ? error.message : "Unknown error"}`);
+        }
+    }
+    /**
+     * Extract URL preview (paid, requires x402 payment)
+     *
+     * @param url
+     * @throws {InvalidUrlError} if URL is invalid
+     * @throws {PaymentFailedError} if payment fails
+     * @throws {ExtractionFailedError} if extraction fails
+     */
+    async extractUrlPreview(url) {
+        try {
+            // Validate and normalize URL
+            const normalizedUrl = validateAndNormalizeUrl(url);
+            // Build request URL
+            const endpoint = `/api/v1/x402/extract/url-preview?url=${encodeURIComponent(normalizedUrl)}`;
+            const requestUrl = `${this.baseUrl}${endpoint}`;
+            // Make request with x402 payment handling
+            const { response, payment } = await handlePayment(requestUrl, this.config);
+            // Check if extraction succeeded
+            if (!response.ok) {
+                throw new ExtractionFailedError(normalizedUrl, `Preview extraction failed: ${response.status} ${response.statusText}`);
+            }
+            const data = (await response.json());
+            return {
+                success: data.success,
+                results: data.results,
+                payment,
+            };
+        }
+        catch (error) {
+            if (error instanceof InvalidUrlError ||
+                error instanceof ExtractionFailedError ||
+                error instanceof PaymentFailedError ||
+                error instanceof NetworkError) {
+                throw error;
+            }
+            throw new ExtractionFailedError(url, `Preview extraction failed: ${error instanceof Error ? error.message : "Unknown error"}`);
+        }
+    }
+    /**
+     * Extract URL content as markdown (paid, requires x402 payment)
+     *
+     * @param url
+     * @param options
+     * @param options.includeMediaUrls
+     * @throws {InvalidUrlError} if URL is invalid
+     * @throws {PaymentFailedError} if payment fails
+     * @throws {ExtractionFailedError} if extraction fails
+     */
+    async extractUrlContent(url, options) {
+        try {
+            // Validate and normalize URL
+            const normalizedUrl = validateAndNormalizeUrl(url);
+            // Build request URL with optional params
+            const params = new URLSearchParams({ url: normalizedUrl });
+            if (options?.includeMediaUrls) {
+                params.set("includeMediaUrls", "true");
+            }
+            const endpoint = `/api/v1/x402/extract/url-content?${params.toString()}`;
+            const requestUrl = `${this.baseUrl}${endpoint}`;
+            // Make request with x402 payment handling
+            const { response, payment } = await handlePayment(requestUrl, this.config);
+            // Check if extraction succeeded
+            if (!response.ok) {
+                throw new ExtractionFailedError(normalizedUrl, `Content extraction failed: ${response.status} ${response.statusText}`);
+            }
+            const data = (await response.json());
+            return {
+                success: data.success,
+                results: data.results,
+                payment,
+            };
+        }
+        catch (error) {
+            if (error instanceof InvalidUrlError ||
+                error instanceof ExtractionFailedError ||
+                error instanceof PaymentFailedError ||
+                error instanceof NetworkError) {
+                throw error;
+            }
+            throw new ExtractionFailedError(url, `Content extraction failed: ${error instanceof Error ? error.message : "Unknown error"}`);
+        }
+    }
+    /**
+     * Check URL and extract metadata in one call
+     * Throws RobotsBlockedError if robots.txt blocks the URL
+     *
+     * @param url
+     * @param options
+     * @param options.includeResponseBody
+     */
+    async checkAndExtractUrlMetadata(url, options) {
+        const checkResponse = await this.preflightUrlCheck(url);
+        if (!checkResponse.results[0]?.data?.allowed) {
+            throw new RobotsBlockedError(url, checkResponse.results[0]?.data?.message || "URL is blocked by robots.txt");
+        }
+        return this.extractUrlMetadata(url, options);
+    }
+    /**
+     * Check URL and extract links in one call
+     * Throws RobotsBlockedError if robots.txt blocks the URL
+     *
+     * @param url
+     */
+    async checkAndExtractUrlLinks(url) {
+        const checkResponse = await this.preflightUrlCheck(url);
+        if (!checkResponse.results[0]?.data?.allowed) {
+            throw new RobotsBlockedError(url, checkResponse.results[0]?.data?.message || "URL is blocked by robots.txt");
+        }
+        return this.extractUrlLinks(url);
+    }
+    /**
+     * Check URL and extract preview in one call
+     * Throws RobotsBlockedError if robots.txt blocks the URL
+     *
+     * @param url
+     */
+    async checkAndExtractUrlPreview(url) {
+        const checkResponse = await this.preflightUrlCheck(url);
+        if (!checkResponse.results[0]?.data?.allowed) {
+            throw new RobotsBlockedError(url, checkResponse.results[0]?.data?.message || "URL is blocked by robots.txt");
+        }
+        return this.extractUrlPreview(url);
+    }
+    /**
+     * Check URL and extract content in one call
+     * Throws RobotsBlockedError if robots.txt blocks the URL
+     *
+     * @param url
+     * @param options
+     * @param options.includeMediaUrls
+     */
+    async checkAndExtractUrlContent(url, options) {
+        const checkResponse = await this.preflightUrlCheck(url);
+        if (!checkResponse.results[0]?.data?.allowed) {
+            throw new RobotsBlockedError(url, checkResponse.results[0]?.data?.message || "URL is blocked by robots.txt");
+        }
+        return this.extractUrlContent(url, options);
+    }
+}

package/dist/esm/index.js

@@ -0,0 +1,2 @@
+export { MinifetchClient } from "./client.js";
+export { MinifetchClient as default } from "./client.js";

package/dist/esm/init.js

@@ -0,0 +1,70 @@
+import { VALID_NETWORKS } from "./types/config.js";
+import { ConfigurationError } from "./types/errors.js";
+/**
+ * Default configuration values
+ * Exported for testing purposes
+ */
+export const DEFAULTS = {
+    network: "base",
+    apiBaseUrls: {
+        "base-sepolia": "http://localhost:4021",
+        "solana-devnet": "http://localhost:4021",
+        base: "https://minifetch.com",
+        solana: "https://minifetch.com",
+    },
+    explorerUrls: {
+        "base-sepolia": "https://sepolia.basescan.org/tx",
+        "solana-devnet": "https://explorer.solana.com/tx?cluster=devnet",
+        base: "https://basescan.org/tx",
+        solana: "https://explorer.solana.com/tx",
+    },
+};
+/**
+ * Validate & initialize client configuration
+ *
+ * @param config
+ */
+export function initConfig(config) {
+    // Validate private key is provided
+    if (!config.privateKey || config.privateKey.trim() === "") {
+        throw new ConfigurationError("Private key is required");
+    }
+    // Validate network
+    const network = config.network || DEFAULTS.network;
+    if (!VALID_NETWORKS.includes(network)) {
+        throw new ConfigurationError(`Invalid network: "${network}". Must be one of: ${VALID_NETWORKS.join(", ")}`);
+    }
+    // Validate private key format based on network
+    validatePrivateKey(config.privateKey, network);
+    // Build initalized config
+    const apiBaseUrl = DEFAULTS.apiBaseUrls[network];
+    const explorerUrl = DEFAULTS.explorerUrls[network];
+    // console.log(`config.ts: network: ${network} apiBaseUrl: ${apiBaseUrl}`)
+    return {
+        network,
+        privateKey: config.privateKey,
+        apiBaseUrl,
+        explorerUrl,
+    };
+}
+/**
+ * Validate private key format for the given network
+ *
+ * @param privateKey
+ * @param network
+ */
+function validatePrivateKey(privateKey, network) {
+    if (network === "solana" || network === "solana-devnet") {
+        // Solana: base58 encoded, typically 88 characters
+        if (!/^[1-9A-HJ-NP-Za-km-z]{87,88}$/.test(privateKey)) {
+            throw new ConfigurationError("Invalid Solana private key format (expected base58 string)");
+        }
+    }
+    else {
+        // EVM: hex string, 64 characters (with or without 0x prefix)
+        const cleanKey = privateKey.startsWith("0x") ? privateKey.slice(2) : privateKey;
+        if (!/^[0-9a-fA-F]{64}$/.test(cleanKey)) {
+            throw new ConfigurationError("Invalid EVM private key format (expected hex string that starts with 0x)");
+        }
+    }
+}

package/dist/esm/types/config.js

@@ -0,0 +1 @@
+export const VALID_NETWORKS = ["base", "base-sepolia", "solana", "solana-devnet"];

package/dist/esm/types/errors.js

@@ -0,0 +1,122 @@
+/**
+ * Base error class for all Minifetch errors
+ */
+export class MinifetchError extends Error {
+    /**
+     *
+     * @param message
+     */
+    constructor(message) {
+        super(message);
+        this.name = "MinifetchError";
+        Object.setPrototypeOf(this, MinifetchError.prototype);
+    }
+}
+/**
+ * Thrown when URL validation fails
+ */
+export class InvalidUrlError extends MinifetchError {
+    url;
+    /**
+     *
+     * @param url
+     * @param message
+     */
+    constructor(url, message) {
+        super(message || `Invalid url: ${url}`);
+        this.name = "InvalidUrlError";
+        this.url = url;
+        Object.setPrototypeOf(this, InvalidUrlError.prototype);
+    }
+}
+/**
+ * Thrown when robots.txt blocks the request
+ */
+export class RobotsBlockedError extends MinifetchError {
+    url;
+    /**
+     *
+     * @param url
+     * @param message
+     */
+    constructor(url, message) {
+        super(message || `URL is blocked by robots.txt`);
+        this.name = "RobotsBlockedError";
+        this.url = url;
+        Object.setPrototypeOf(this, RobotsBlockedError.prototype);
+    }
+}
+/**
+ * Thrown when payment fails
+ */
+export class PaymentFailedError extends MinifetchError {
+    network;
+    originalError;
+    /**
+     *
+     * @param message
+     * @param network
+     * @param originalError
+     */
+    constructor(message, network, originalError) {
+        super(message);
+        this.name = "PaymentFailedError";
+        this.network = network;
+        this.originalError = originalError;
+        Object.setPrototypeOf(this, PaymentFailedError.prototype);
+    }
+}
+/**
+ * Thrown when extraction/fetch fails
+ */
+export class ExtractionFailedError extends MinifetchError {
+    url;
+    statusCode;
+    originalError;
+    /**
+     *
+     * @param url
+     * @param message
+     * @param statusCode
+     * @param originalError
+     */
+    constructor(url, message, statusCode, originalError) {
+        super(message);
+        this.name = "ExtractionFailedError";
+        this.url = url;
+        this.statusCode = statusCode;
+        this.originalError = originalError;
+        Object.setPrototypeOf(this, ExtractionFailedError.prototype);
+    }
+}
+/**
+ * Thrown when configuration is invalid
+ */
+export class ConfigurationError extends MinifetchError {
+    /**
+     *
+     * @param message
+     */
+    constructor(message) {
+        super(message);
+        this.name = "ConfigurationError";
+        Object.setPrototypeOf(this, ConfigurationError.prototype);
+    }
+}
+/**
+ * Thrown when network/API communication fails
+ */
+export class NetworkError extends MinifetchError {
+    originalError;
+    /**
+     *
+     * @param message
+     * @param originalError
+     */
+    constructor(message, originalError) {
+        super(message);
+        this.name = "NetworkError";
+        this.originalError = originalError;
+        Object.setPrototypeOf(this, NetworkError.prototype);
+    }
+}

package/dist/esm/types/responses.js

@@ -0,0 +1 @@
+export {};