@vakra-dev/reader-js 0.2.0

package/README.md

@@ -0,0 +1,107 @@
+# @vakra-dev/reader-js
+
+TypeScript/JavaScript SDK for the [Reader API](https://reader.dev) — content extraction for LLMs. Wraps `POST /v1/read`, parses the standard envelope, throws typed errors, and auto-polls async jobs to completion.
+
+**Version:** 0.2.0 · **Runtime:** Node 18+, Deno, Bun, Cloudflare Workers, modern browsers
+
+## Install
+
+```bash
+npm install @vakra-dev/reader-js
+```
+
+## Quick start
+
+```ts
+import { ReaderClient } from "@vakra-dev/reader-js";
+
+const reader = new ReaderClient({ apiKey: process.env.READER_KEY! });
+
+const result = await reader.read({ url: "https://example.com" });
+if (result.kind === "scrape") {
+  console.log(result.data.markdown);
+}
+```
+
+`reader.read(...)` returns a discriminated union:
+
+- `{ kind: "scrape", data: ScrapeResult }` — single-URL requests, returned immediately
+- `{ kind: "job", data: Job }` — batch and crawl requests, auto-polled to completion
+
+## Features
+
+- **One method for every read operation.** `reader.read({ url })` for sync scrape, `{ urls: [...] }` for batch, `{ url, maxPages }` for crawl.
+- **Typed errors for all 11 Reader error codes.** `InsufficientCreditsError`, `RateLimitedError`, `UrlBlockedError`, `ScrapeTimeoutError`, and more. Each subclass surfaces the relevant fields (e.g. `err.required`, `err.retryAfterSeconds`).
+- **Automatic retries with exponential backoff** for transient codes (`rate_limited`, `upstream_unavailable`, `scrape_timeout`, …). Honors the `Retry-After` header on 429.
+- **Pagination-aware job collection.** `waitForJob()` returns the full job with every page result collected across pagination boundaries.
+- **SSE streaming.** `for await (const event of reader.stream(jobId))` yields real-time `progress` / `page` / `error` / `done` events.
+- **Request ID tracing.** Every error carries the `x-request-id` header value on `err.requestId` for support tickets.
+
+## Browser Sessions
+
+Launch a stealthed Chrome and connect Playwright or Puppeteer:
+
+```ts
+import { chromium } from "playwright-core";
+
+const session = await reader.sessions.create();
+const browser = await chromium.connectOverCDP(session.wsEndpoint);
+const page = await (await browser.newContext()).newPage();
+
+await page.goto("https://example.com");
+console.log(await page.title());
+
+await browser.close();
+await reader.sessions.stop(session.sessionId);
+```
+
+Methods: `reader.sessions.create()`, `.get(id)`, `.stop(id)`, `.list()`
+
+## Browser usage
+
+The SDK works in modern browsers via native `fetch`, but **do not ship your API key in browser code** — anyone can read and reuse it. Proxy requests through your own backend.
+
+## Errors
+
+```ts
+import {
+  ReaderApiError,
+  InsufficientCreditsError,
+  RateLimitedError,
+  UrlBlockedError,
+} from "@vakra-dev/reader-js";
+
+try {
+  await reader.read({ url });
+} catch (err) {
+  if (err instanceof InsufficientCreditsError) {
+    console.error(`Need ${err.required}, have ${err.available}`);
+  } else if (err instanceof RateLimitedError) {
+    console.error(`Retry after ${err.retryAfterSeconds}s`);
+  } else if (err instanceof UrlBlockedError) {
+    console.error(`Blocked: ${err.reason}`);
+  } else if (err instanceof ReaderApiError) {
+    console.error(`[${err.code}] ${err.message} — see ${err.docsUrl}`);
+  } else {
+    throw err;
+  }
+}
+```
+
+Full catalog of error codes: https://reader.dev/docs/home/concepts/errors
+
+## Links
+
+- **Docs:** https://reader.dev/docs
+- **SDK reference:** https://reader.dev/docs/sdk/javascript
+- **API reference:** https://reader.dev/docs/api-reference/read
+- **Discord:** https://discord.gg/6tjkq7J5WV
+
+## Development
+
+```bash
+npm install
+npm run typecheck
+npm run build    # builds to dist/
+npm test         # vitest
+```

package/dist/README.md

@@ -0,0 +1,107 @@
+# @vakra-dev/reader-js
+
+TypeScript/JavaScript SDK for the [Reader API](https://reader.dev) — content extraction for LLMs. Wraps `POST /v1/read`, parses the standard envelope, throws typed errors, and auto-polls async jobs to completion.
+
+**Version:** 0.2.0 · **Runtime:** Node 18+, Deno, Bun, Cloudflare Workers, modern browsers
+
+## Install
+
+```bash
+npm install @vakra-dev/reader-js
+```
+
+## Quick start
+
+```ts
+import { ReaderClient } from "@vakra-dev/reader-js";
+
+const reader = new ReaderClient({ apiKey: process.env.READER_KEY! });
+
+const result = await reader.read({ url: "https://example.com" });
+if (result.kind === "scrape") {
+  console.log(result.data.markdown);
+}
+```
+
+`reader.read(...)` returns a discriminated union:
+
+- `{ kind: "scrape", data: ScrapeResult }` — single-URL requests, returned immediately
+- `{ kind: "job", data: Job }` — batch and crawl requests, auto-polled to completion
+
+## Features
+
+- **One method for every read operation.** `reader.read({ url })` for sync scrape, `{ urls: [...] }` for batch, `{ url, maxPages }` for crawl.
+- **Typed errors for all 11 Reader error codes.** `InsufficientCreditsError`, `RateLimitedError`, `UrlBlockedError`, `ScrapeTimeoutError`, and more. Each subclass surfaces the relevant fields (e.g. `err.required`, `err.retryAfterSeconds`).
+- **Automatic retries with exponential backoff** for transient codes (`rate_limited`, `upstream_unavailable`, `scrape_timeout`, …). Honors the `Retry-After` header on 429.
+- **Pagination-aware job collection.** `waitForJob()` returns the full job with every page result collected across pagination boundaries.
+- **SSE streaming.** `for await (const event of reader.stream(jobId))` yields real-time `progress` / `page` / `error` / `done` events.
+- **Request ID tracing.** Every error carries the `x-request-id` header value on `err.requestId` for support tickets.
+
+## Browser Sessions
+
+Launch a stealthed Chrome and connect Playwright or Puppeteer:
+
+```ts
+import { chromium } from "playwright-core";
+
+const session = await reader.sessions.create();
+const browser = await chromium.connectOverCDP(session.wsEndpoint);
+const page = await (await browser.newContext()).newPage();
+
+await page.goto("https://example.com");
+console.log(await page.title());
+
+await browser.close();
+await reader.sessions.stop(session.sessionId);
+```
+
+Methods: `reader.sessions.create()`, `.get(id)`, `.stop(id)`, `.list()`
+
+## Browser usage
+
+The SDK works in modern browsers via native `fetch`, but **do not ship your API key in browser code** — anyone can read and reuse it. Proxy requests through your own backend.
+
+## Errors
+
+```ts
+import {
+  ReaderApiError,
+  InsufficientCreditsError,
+  RateLimitedError,
+  UrlBlockedError,
+} from "@vakra-dev/reader-js";
+
+try {
+  await reader.read({ url });
+} catch (err) {
+  if (err instanceof InsufficientCreditsError) {
+    console.error(`Need ${err.required}, have ${err.available}`);
+  } else if (err instanceof RateLimitedError) {
+    console.error(`Retry after ${err.retryAfterSeconds}s`);
+  } else if (err instanceof UrlBlockedError) {
+    console.error(`Blocked: ${err.reason}`);
+  } else if (err instanceof ReaderApiError) {
+    console.error(`[${err.code}] ${err.message} — see ${err.docsUrl}`);
+  } else {
+    throw err;
+  }
+}
+```
+
+Full catalog of error codes: https://reader.dev/docs/home/concepts/errors
+
+## Links
+
+- **Docs:** https://reader.dev/docs
+- **SDK reference:** https://reader.dev/docs/sdk/javascript
+- **API reference:** https://reader.dev/docs/api-reference/read
+- **Discord:** https://discord.gg/6tjkq7J5WV
+
+## Development
+
+```bash
+npm install
+npm run typecheck
+npm run build    # builds to dist/
+npm test         # vitest
+```

package/dist/index.cjs

@@ -0,0 +1,500 @@
+"use strict";
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/index.ts
+var index_exports = {};
+__export(index_exports, {
+  ConcurrencyLimitedError: () => ConcurrencyLimitedError,
+  ConflictError: () => ConflictError,
+  InsufficientCreditsError: () => InsufficientCreditsError,
+  InternalServerError: () => InternalServerError,
+  InvalidRequestError: () => InvalidRequestError,
+  NotFoundError: () => NotFoundError,
+  RateLimitedError: () => RateLimitedError,
+  ReaderApiError: () => ReaderApiError,
+  ReaderClient: () => ReaderClient,
+  ScrapeTimeoutError: () => ScrapeTimeoutError,
+  UnauthenticatedError: () => UnauthenticatedError,
+  UpstreamUnavailableError: () => UpstreamUnavailableError,
+  UrlBlockedError: () => UrlBlockedError,
+  toReaderApiError: () => toReaderApiError
+});
+module.exports = __toCommonJS(index_exports);
+
+// src/errors.ts
+var ReaderApiError = class extends Error {
+  code;
+  httpStatus;
+  details;
+  docsUrl;
+  requestId;
+  constructor(body, httpStatus, requestId) {
+    super(body.message);
+    this.name = "ReaderApiError";
+    this.code = body.code;
+    this.httpStatus = httpStatus;
+    this.details = body.details;
+    this.docsUrl = body.docsUrl;
+    this.requestId = requestId;
+  }
+};
+var InvalidRequestError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "InvalidRequestError";
+  }
+};
+var UnauthenticatedError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "UnauthenticatedError";
+  }
+};
+var InsufficientCreditsError = class extends ReaderApiError {
+  required;
+  available;
+  resetAt;
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "InsufficientCreditsError";
+    this.required = body.details?.required;
+    this.available = body.details?.available;
+    this.resetAt = body.details?.resetAt;
+  }
+};
+var UrlBlockedError = class extends ReaderApiError {
+  url;
+  reason;
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "UrlBlockedError";
+    this.url = body.details?.url;
+    this.reason = body.details?.reason;
+  }
+};
+var NotFoundError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "NotFoundError";
+  }
+};
+var ConflictError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "ConflictError";
+  }
+};
+var RateLimitedError = class extends ReaderApiError {
+  retryAfterSeconds;
+  limit;
+  windowSeconds;
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "RateLimitedError";
+    this.retryAfterSeconds = body.details?.retryAfterSeconds;
+    this.limit = body.details?.limit;
+    this.windowSeconds = body.details?.windowSeconds;
+  }
+};
+var ConcurrencyLimitedError = class extends ReaderApiError {
+  active;
+  max;
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "ConcurrencyLimitedError";
+    this.active = body.details?.active;
+    this.max = body.details?.max;
+  }
+};
+var InternalServerError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "InternalServerError";
+  }
+};
+var UpstreamUnavailableError = class extends ReaderApiError {
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "UpstreamUnavailableError";
+  }
+};
+var ScrapeTimeoutError = class extends ReaderApiError {
+  timeoutMs;
+  constructor(body, status, requestId) {
+    super(body, status, requestId);
+    this.name = "ScrapeTimeoutError";
+    this.timeoutMs = body.details?.timeoutMs;
+  }
+};
+function toReaderApiError(body, httpStatus, requestId) {
+  switch (body.code) {
+    case "invalid_request":
+      return new InvalidRequestError(body, httpStatus, requestId);
+    case "unauthenticated":
+      return new UnauthenticatedError(body, httpStatus, requestId);
+    case "insufficient_credits":
+      return new InsufficientCreditsError(body, httpStatus, requestId);
+    case "url_blocked":
+      return new UrlBlockedError(body, httpStatus, requestId);
+    case "not_found":
+      return new NotFoundError(body, httpStatus, requestId);
+    case "conflict":
+      return new ConflictError(body, httpStatus, requestId);
+    case "rate_limited":
+      return new RateLimitedError(body, httpStatus, requestId);
+    case "concurrency_limited":
+      return new ConcurrencyLimitedError(body, httpStatus, requestId);
+    case "internal_error":
+      return new InternalServerError(body, httpStatus, requestId);
+    case "upstream_unavailable":
+      return new UpstreamUnavailableError(body, httpStatus, requestId);
+    case "scrape_timeout":
+      return new ScrapeTimeoutError(body, httpStatus, requestId);
+    default:
+      return new ReaderApiError(body, httpStatus, requestId);
+  }
+}
+
+// src/client.ts
+var DEFAULT_BASE_URL = "https://api.reader.dev";
+var DEFAULT_TIMEOUT = 6e4;
+var DEFAULT_MAX_RETRIES = 2;
+var DEFAULT_POLL_INTERVAL = 2e3;
+var DEFAULT_POLL_TIMEOUT = 3e5;
+var ReaderClient = class {
+  apiKey;
+  baseUrl;
+  timeout;
+  maxRetries;
+  extraHeaders;
+  _sessions = null;
+  constructor(config) {
+    if (!config.apiKey) {
+      throw new Error("API key is required");
+    }
+    this.apiKey = config.apiKey;
+    this.baseUrl = (config.baseUrl || DEFAULT_BASE_URL).replace(/\/$/, "");
+    this.timeout = config.timeout || DEFAULT_TIMEOUT;
+    this.maxRetries = config.maxRetries ?? DEFAULT_MAX_RETRIES;
+    this.extraHeaders = config.headers || {};
+  }
+  /**
+   * Browser sessions API.
+   *
+   * @example
+   * ```typescript
+   * const session = await client.sessions.create();
+   * const browser = await chromium.connectOverCDP(session.wsEndpoint);
+   * // ... use Playwright ...
+   * await client.sessions.stop(session.sessionId);
+   * ```
+   */
+  get sessions() {
+    if (!this._sessions) {
+      this._sessions = new SessionsAPI(this.request.bind(this));
+    }
+    return this._sessions;
+  }
+  /**
+   * Read (scrape, batch, or crawl) one or more URLs.
+   *
+   * - Single URL → sync scrape, returns immediately with `{ kind: "scrape", data }`
+   * - Multiple URLs or URL + maxDepth/maxPages → async job; this method polls
+   *   until the job terminates and returns `{ kind: "job", data }`.
+   */
+  async read(params) {
+    const envelope = await this.request(
+      "POST",
+      "/v1/read",
+      params
+    );
+    const data = envelope.data;
+    if (data && typeof data === "object" && "status" in data && "mode" in data && !("markdown" in data) && !("metadata" in data)) {
+      const jobId = String(data.id);
+      const job = await this.waitForJob(jobId);
+      return { kind: "job", data: job };
+    }
+    return { kind: "scrape", data };
+  }
+  /**
+   * Get job status and a single page of results.
+   */
+  async getJob(jobId, opts) {
+    const query = new URLSearchParams();
+    if (opts?.skip !== void 0) query.set("skip", String(opts.skip));
+    if (opts?.limit !== void 0) query.set("limit", String(opts.limit));
+    const qs = query.toString();
+    const envelope = await this.request(
+      "GET",
+      `/v1/jobs/${jobId}${qs ? `?${qs}` : ""}`
+    );
+    return {
+      job: envelope.data,
+      hasMore: envelope.pagination.hasMore,
+      next: envelope.pagination.next
+    };
+  }
+  /**
+   * Fetch all job result pages by following pagination.
+   */
+  async getAllJobResults(jobId) {
+    const pages = [];
+    let skip = 0;
+    const limit = 100;
+    while (true) {
+      const { job, hasMore } = await this.getJob(jobId, { skip, limit });
+      pages.push(...job.results ?? []);
+      if (!hasMore) break;
+      skip += limit;
+    }
+    return pages;
+  }
+  /**
+   * Cancel a job. Throws `ConflictError` if the job is already terminal.
+   */
+  async cancelJob(jobId) {
+    await this.request("DELETE", `/v1/jobs/${jobId}`);
+  }
+  /**
+   * Retry the failed URLs in a job. Throws `InvalidRequestError` if no
+   * failed URLs exist.
+   */
+  async retryJob(jobId) {
+    const envelope = await this.request("POST", `/v1/jobs/${jobId}/retry`);
+    return envelope.data;
+  }
+  /**
+   * Poll a job until it completes, fails, or is cancelled. Collects all
+   * paginated results when complete.
+   */
+  async waitForJob(jobId, options) {
+    const interval = options?.pollInterval ?? DEFAULT_POLL_INTERVAL;
+    const timeout = options?.timeout ?? DEFAULT_POLL_TIMEOUT;
+    const start = Date.now();
+    while (Date.now() - start < timeout) {
+      const { job } = await this.getJob(jobId, { limit: 1 });
+      if (job.status === "completed" || job.status === "failed" || job.status === "cancelled") {
+        if (job.status === "completed") {
+          job.results = await this.getAllJobResults(jobId);
+        }
+        return job;
+      }
+      await sleep(interval);
+    }
+    throw new ScrapeTimeoutError(
+      {
+        code: "scrape_timeout",
+        message: `Job ${jobId} polling timed out after ${timeout}ms`,
+        details: { timeoutMs: timeout }
+      },
+      504
+    );
+  }
+  /**
+   * Stream job results as they arrive via polling.
+   *
+   * @example
+   * for await (const event of client.stream(jobId)) {
+   *   if (event.type === "page") console.log(event.data.url);
+   *   if (event.type === "done") break;
+   * }
+   */
+  async *stream(jobId, options) {
+    const interval = options?.pollInterval ?? DEFAULT_POLL_INTERVAL;
+    const timeout = options?.timeout ?? DEFAULT_POLL_TIMEOUT;
+    const start = Date.now();
+    let lastCompleted = 0;
+    while (Date.now() - start < timeout) {
+      const { job } = await this.getJob(jobId, { skip: lastCompleted, limit: 100 });
+      yield {
+        type: "progress",
+        completed: job.completed,
+        total: job.total,
+        status: job.status
+      };
+      for (const page of job.results ?? []) {
+        if (page.error) {
+          yield { type: "error", url: page.url, error: page.error };
+        } else {
+          yield { type: "page", data: page };
+        }
+        lastCompleted += 1;
+      }
+      if (job.status === "completed" || job.status === "failed" || job.status === "cancelled") {
+        yield {
+          type: "done",
+          completed: job.completed,
+          total: job.total,
+          status: job.status
+        };
+        return;
+      }
+      await sleep(interval);
+    }
+    throw new ScrapeTimeoutError(
+      {
+        code: "scrape_timeout",
+        message: `Job ${jobId} stream timed out`,
+        details: { timeoutMs: timeout }
+      },
+      504
+    );
+  }
+  /**
+   * Get the current credit balance for this workspace.
+   */
+  async getCredits() {
+    const envelope = await this.request("GET", "/v1/usage/credits");
+    return envelope.data;
+  }
+  // --- Internal ---
+  async request(method, path, body) {
+    const url = path.startsWith("http") ? path : `${this.baseUrl}${path}`;
+    let lastError = null;
+    for (let attempt = 0; attempt <= this.maxRetries; attempt++) {
+      try {
+        const controller = new AbortController();
+        const timeoutId = setTimeout(() => controller.abort(), this.timeout);
+        const res = await fetch(url, {
+          method,
+          headers: {
+            "Content-Type": "application/json",
+            "x-api-key": this.apiKey,
+            ...this.extraHeaders
+          },
+          body: body ? JSON.stringify(body) : void 0,
+          signal: controller.signal
+        });
+        clearTimeout(timeoutId);
+        const requestId = res.headers.get("x-request-id") ?? void 0;
+        const parsed = await res.json().catch(() => null);
+        if (!res.ok) {
+          if (parsed && "error" in parsed && parsed.error) {
+            const err = toReaderApiError(parsed.error, res.status, requestId);
+            if (res.status < 500 && res.status !== 429) throw err;
+            if (err instanceof RateLimitedError && err.retryAfterSeconds) {
+              await sleep(err.retryAfterSeconds * 1e3);
+            }
+            lastError = err;
+          } else {
+            const genericErr = new ReaderApiError(
+              {
+                code: "internal_error",
+                message: `Request failed with status ${res.status}`
+              },
+              res.status,
+              requestId
+            );
+            if (res.status < 500) throw genericErr;
+            lastError = genericErr;
+          }
+        } else {
+          return parsed;
+        }
+      } catch (err) {
+        if (err instanceof ReaderApiError) {
+          if (err.httpStatus < 500 && err.httpStatus !== 429) throw err;
+          lastError = err;
+        } else if (err instanceof Error) {
+          if (err.name === "AbortError") {
+            lastError = new ReaderApiError(
+              { code: "scrape_timeout", message: "Request timed out" },
+              504
+            );
+          } else {
+            lastError = err;
+          }
+        }
+      }
+      if (attempt < this.maxRetries) {
+        await sleep(Math.pow(2, attempt) * 1e3);
+      }
+    }
+    throw lastError ?? new ReaderApiError({ code: "internal_error", message: "Request failed" }, 500);
+  }
+};
+var SessionsAPI = class {
+  constructor(request) {
+    this.request = request;
+  }
+  request;
+  /**
+   * Create a browser session. Returns a CDP WebSocket URL for
+   * Playwright/Puppeteer connection.
+   */
+  async create(params) {
+    const envelope = await this.request(
+      "POST",
+      "/v1/sessions",
+      params ?? {}
+    );
+    return envelope.data;
+  }
+  /**
+   * Get session status.
+   */
+  async get(sessionId) {
+    const envelope = await this.request(
+      "GET",
+      `/v1/sessions/${sessionId}`
+    );
+    return envelope.data;
+  }
+  /**
+   * Stop a browser session.
+   */
+  async stop(sessionId) {
+    const envelope = await this.request(
+      "DELETE",
+      `/v1/sessions/${sessionId}`
+    );
+    return envelope.data;
+  }
+  /**
+   * List active sessions.
+   */
+  async list() {
+    const envelope = await this.request(
+      "GET",
+      "/v1/sessions"
+    );
+    return envelope.data;
+  }
+};
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  ConcurrencyLimitedError,
+  ConflictError,
+  InsufficientCreditsError,
+  InternalServerError,
+  InvalidRequestError,
+  NotFoundError,
+  RateLimitedError,
+  ReaderApiError,
+  ReaderClient,
+  ScrapeTimeoutError,
+  UnauthenticatedError,
+  UpstreamUnavailableError,
+  UrlBlockedError,
+  toReaderApiError
+});
+//# sourceMappingURL=index.cjs.map