thordata 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+- Added an independent SERP client and raw response access.
+- Added task creation, polling, downloading, and status results for seven Web Scraper products.
+- Fixed rawSearch engine normalization, cancellation errors, and Web Scraper parameter messages.
+- Added cross-language contract fixture checks and non-zero example failure exit codes.
+- Added request ID fallback for bare SERP results, business error classification, and parameter messages.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thordata
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,95 @@
-# Temporary Holding Version
+# Thordata JavaScript SDK
+
+JavaScript/Node.js client for the Thordata SERP API and seven verified Web Scraper products. The package is independent of LangChain and requires Node.js 18 or newer.
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Turn live web data into reliable Node.js applications faster. Thordata JavaScript SDK gives developers one interface for real-time search and structured web scraping, making it easier to build research tools, monitoring services, data pipelines, RAG systems, and AI agents.
+
+Use it to track search visibility, enrich business and product data, monitor markets, or bring fresh web data into JavaScript and TypeScript applications without maintaining a separate scraper for every target.
+
+## Why Thordata JavaScript SDK
+
+- **Search and scrape through one client**: Query live search engines and collect structured records through ready-to-use Web Scraper actions.
+- **Built for Node.js workflows**: Add web intelligence to backend services, scheduled jobs, automation, and AI applications with a familiar JavaScript interface.
+- **Simple async integration**: Use promises and standard Node.js patterns for search and structured data collection.
+- **Ready for downstream systems**: Pass structured results into analytics, enrichment, RAG, agents, and internal applications.
+- **Designed for dependable use**: Benefit from predictable response handling, bounded task polling, and redacted diagnostics in production workflows.
+
+Create a Thordata account to get started:
+
+- [Get started with Thordata](https://www.thordata.com/?ls=github&lk=thordata)
+- [Read the official documentation](https://doc.thordata.com)
+
+## Install
+
+```bash
+npm install thordata
+```
+
+Set `THORDATA_API_KEY`, `THORDATA_API_TOKEN`, or the compatible `TALORDATA_API_TOKEN` environment variable. `apiKey`, `apiToken`, or `token` constructor options take precedence.
+
+## SERP
+
+```js
+const { Client } = require("thordata");
+
+async function main() {
+  const client = new Client();
+  const result = await client.search({
+    engine: "google_search",
+    q: "latest AI agent research",
+    gl: "us",
+    hl: "en",
+  });
+  console.log(result);
+}
+
+main().catch(console.error);
+```
+
+Requests are form-encoded `POST` calls to `https://scraperapi.thordata.com/request` with Bearer authentication. SERP source headers are `Origin: JavaScript`, `platform: javascript`, and `api-source: sdk`. The client supports the verified Thordata engines and API-native parameters. `google_search` maps to `google`, `google_places` maps to `google_local`, and `query`, `country`, and `language` map to `q`, `gl`, and `hl`. Yandex sends `q` as `text`. Boolean values become `1` or `0`, and unknown fields are ignored.
+
+Use `searchResponse()` for task metadata and the raw response envelope, and `rawSearch()` for the unparsed response body. `taskId` comes from the upstream envelope when present and falls back to `search_metadata.id` for bare SERP results. The raw response keeps the engine-specific shape; Google commonly returns `organic`/`images`/`places`, while other engines may return `organic_results` or similar fields. `rawSearch()` returns text without interpreting business error codes. The SDK does not expose `searchHtml()` because the current `/request` endpoint does not guarantee HTML.
+
+## Web Scraper
+
+The initial SDK supports these spider IDs:
+
+- `linkedin_company_information_by-url`: `url`
+- `linkedin_job_listings_information_by-job-url`: `job_url`
+- `youtube_video-post_by-keyword`: `keyword`, optional `num_of_posts`
+- `facebook_profile_by-profiles-url`: `url`
+- `youtube_video-post_by-explore`: `url`, optional `all_tabs`
+- `twitter_profile_by-profileurl`: `url`
+- `amazon_product_by-category-url`: `url`, `page_turning`, optional `sort_by`
+
+```js
+const { Client } = require("thordata");
+
+const client = new Client();
+const records = await client.scrape(
+  "linkedin_company_information_by-url",
+  { url: "https://www.linkedin.com/company/microsoft" },
+);
+console.log(records);
+```
+
+`scrape()` submits one task, polls every five seconds, and downloads final JSON records. Web Scraper source fields are form parameters `api_source=sdk` and `source_platform=javascript`; they are not request headers. Amazon uses `Best Sellers` when `sort_by` is omitted. The total wait is capped at 300 seconds by default. Use `scrapeResponse()` for `taskId`, `spiderId`, `status`, `recordCount`, and `errorRecordCount`. Status is `completed`, `partial`, or `empty`. Pass an `AbortSignal` to cancel local waiting.
+
+For API errors, `code` is the Thordata business code and `statusCode` is the HTTP status. Business errors can therefore have `statusCode: 200`.
+
+Web Scraper errors retain their stage and task ID when safe, while API keys and signed URLs are redacted from error text and diagnostic data.
+
+## Local checks
+
+```bash
+npm test
+npm run check
+```
+
+Tests use a mocked `fetch` and do not consume Thordata credits. See `examples/` for runnable SERP and Web Scraper examples.
+
+## Start Building
+
+Start with a SERP query for live web intelligence, add a Web Scraper action when you need structured records, and connect the results to the systems your team already uses. The same SDK can support a quick prototype today and a repeatable Node.js data pipeline tomorrow.
+
+[Explore Thordata SERP API](https://www.thordata.com/products/serp-api?op=flora&from=github) | [Explore Thordata Web Scraper](https://www.thordata.com/products/web-scraper?op=flora&from=github)

package/examples/basic-search.js

@@ -0,0 +1,20 @@
+"use strict";
+
+const { Client } = require("../src");
+
+async function main() {
+  const client = new Client();
+  const result = await client.search({
+    engine: "google_search",
+    q: "latest AI agent research",
+    gl: "us",
+    hl: "en",
+    num: 5,
+  });
+  console.log(result);
+}
+
+main().catch((error) => {
+  console.error(error);
+  process.exitCode = 1;
+});

package/examples/scrape-basic.js

@@ -0,0 +1,17 @@
+"use strict";
+
+const { Client } = require("../src");
+
+async function main() {
+  const client = new Client();
+  const records = await client.scrape(
+    "linkedin_company_information_by-url",
+    { url: "https://www.linkedin.com/company/microsoft" },
+  );
+  console.log(records);
+}
+
+main().catch((error) => {
+  console.error(error);
+  process.exitCode = 1;
+});

package/examples/specialized-search.js

@@ -0,0 +1,20 @@
+"use strict";
+
+const { Client } = require("../src");
+
+async function main() {
+  const client = new Client();
+  const result = await client.search({
+    engine: "google_flights",
+    departure_id: "SFO",
+    arrival_id: "JFK",
+    outbound_date: "2026-12-01",
+    return_date: "2026-12-08",
+  });
+  console.log(result);
+}
+
+main().catch((error) => {
+  console.error(error);
+  process.exitCode = 1;
+});

package/package.json

@@ -1,6 +1,36 @@
 {
   "name": "thordata",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "JavaScript SDK for the Thordata SERP API and Web Scraper",
+  "author": {
+    "name": "Thordata",
+    "email": "cianna@thordata.com"
+  },
+  "homepage": "https://github.com/Thordata/thordata-javascript#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Thordata/thordata-javascript.git"
+  },
+  "bugs": {
+    "url": "https://github.com/Thordata/thordata-javascript/issues"
+  },
+  "main": "src/index.js",
+  "types": "src/index.d.ts",
+  "type": "commonjs",
+  "files": [
+    "src",
+    "examples",
+    "README.md",
+    "CHANGELOG.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node --test",
+    "check": "node --check src/index.js && node --check src/serp.js && node --check src/web-scraper.js && node --check src/errors.js"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": ["thordata", "serp", "search", "web-scraper", "sdk"],
+  "license": "MIT"
+}

package/src/errors.js

@@ -0,0 +1,10 @@
+"use strict";
+
+class ThorDataError extends Error {
+  constructor(message, options = {}) {
+    super(message, options);
+    this.name = new.target.name;
+  }
+}
+
+module.exports = { ThorDataError };

package/src/index.d.ts

@@ -0,0 +1,198 @@
+export type SearchEngine =
+  | "google"
+  | "google_search"
+  | "google_light"
+  | "google_web"
+  | "google_images"
+  | "google_videos"
+  | "google_news"
+  | "google_shopping"
+  | "google_local"
+  | "google_places"
+  | "google_product"
+  | "google_lens"
+  | "google_trends"
+  | "google_maps"
+  | "google_hotels"
+  | "google_jobs"
+  | "google_scholar"
+  | "google_scholar_cite"
+  | "google_scholar_author"
+  | "google_play"
+  | "google_play_product"
+  | "google_play_games"
+  | "google_play_movies"
+  | "google_play_books"
+  | "google_finance"
+  | "google_finance_markets"
+  | "google_flights"
+  | "google_patents"
+  | "google_patents_details"
+  | "google_ai_mode"
+  | "bing"
+  | "bing_images"
+  | "bing_videos"
+  | "bing_news"
+  | "bing_shopping"
+  | "bing_maps"
+  | "yandex"
+  | "duckduckgo";
+
+export type SearchParams = Record<string, unknown> & {
+  engine?: SearchEngine | string;
+  q?: string;
+  query?: string;
+  gl?: string;
+  country?: string;
+  hl?: string;
+  language?: string;
+  json?: string | number;
+  isjson?: string | number | boolean;
+};
+
+export type WebScraperId =
+  | "linkedin_company_information_by-url"
+  | "linkedin_job_listings_information_by-job-url"
+  | "youtube_video-post_by-keyword"
+  | "facebook_profile_by-profiles-url"
+  | "youtube_video-post_by-explore"
+  | "twitter_profile_by-profileurl"
+  | "amazon_product_by-category-url";
+
+export type WebScraperParams = Record<string, unknown>;
+
+export interface WebScraperOptions {
+  maxWaitSeconds?: number;
+  includeErrors?: boolean;
+  fileName?: string;
+  signal?: AbortSignal;
+}
+
+export interface WebScraperResponse<T = unknown> {
+  result: T;
+  taskId: string;
+  spiderId: WebScraperId;
+  status: "completed" | "partial" | "empty";
+  recordCount: number;
+  errorRecordCount: number;
+  raw: unknown;
+}
+
+export interface SearchResponse<T = unknown> {
+  result: T;
+  taskId: string | null;
+  raw: Record<string, unknown> | unknown;
+}
+
+export interface RequestOptions {
+  headers?: Record<string, string>;
+  signal?: AbortSignal;
+  filterParams?: boolean;
+}
+
+export interface ThorDataClientOptions {
+  apiKey?: string;
+  apiToken?: string;
+  token?: string;
+  apiUrl?: string;
+  baseUrl?: string;
+  timeout?: number;
+  headers?: Record<string, string>;
+}
+
+export declare const DEFAULT_API_URL: string;
+export declare const DEFAULT_TIMEOUT_MS: number;
+export declare const DEFAULT_MAX_WAIT_SECONDS: number;
+export declare const INTEGRATION_PLATFORM: string;
+export declare const INTEGRATION_SOURCE: string;
+export declare const REQUEST_PARAMETERS: ReadonlySet<string>;
+export declare const SUPPORTED_ENGINES: readonly SearchEngine[];
+export declare const SUPPORTED_WEB_SCRAPERS: readonly WebScraperId[];
+
+export declare class ThorDataError extends Error {}
+export declare class ThorDataConfigurationError extends ThorDataError {}
+export declare class ThorDataConnectionError extends ThorDataError {}
+export declare class ThorDataTimeoutError extends ThorDataConnectionError {}
+
+export interface ThorDataAPIErrorOptions {
+  code?: number | null;
+  statusCode?: number | null;
+  responseData?: unknown;
+}
+
+export declare class ThorDataAPIError extends ThorDataError {
+  code: number | null;
+  statusCode: number | null;
+  responseData: unknown;
+  constructor(message: string, options?: ThorDataAPIErrorOptions);
+}
+
+export declare class ThorDataAuthenticationError extends ThorDataAPIError {}
+export declare class ThorDataPermissionError extends ThorDataAPIError {}
+export declare class ThorDataRateLimitError extends ThorDataAPIError {}
+export declare class ThorDataInvalidRequestError extends ThorDataAPIError {}
+export declare class ThorDataNotCollectedError extends ThorDataAPIError {}
+
+export interface ThorDataWebScraperErrorOptions {
+  stage?: string;
+  taskId?: string | null;
+  spiderId?: string | null;
+  reason?: unknown;
+  serviceCode?: string | number | null;
+  secrets?: readonly string[];
+}
+
+export declare class ThorDataWebScraperError extends ThorDataError {
+  errorCode: string;
+  stage: string;
+  taskId: string | null;
+  spiderId: string | null;
+  reason: string | null;
+  serviceCode: string | number | null;
+  constructor(errorCode: string, options?: ThorDataWebScraperErrorOptions);
+}
+
+export declare class ThorDataClient {
+  readonly apiKey: string;
+  readonly apiUrl: string;
+  readonly baseUrl: string;
+  readonly timeout: number;
+  readonly headers: Record<string, string>;
+
+  constructor(options?: ThorDataClientOptions);
+  toString(): string;
+  request(
+    method: string,
+    path?: string,
+    params?: Record<string, unknown>,
+    options?: RequestOptions,
+  ): Promise<Response>;
+  search<T = unknown>(params?: SearchParams, options?: RequestOptions): Promise<T>;
+  searchJson<T = unknown>(params?: SearchParams, options?: RequestOptions): Promise<T>;
+  searchResponse<T = unknown>(params?: SearchParams, options?: RequestOptions): Promise<SearchResponse<T>>;
+  rawSearch(params?: Record<string, unknown>, options?: RequestOptions): Promise<string>;
+  scrape<T = unknown>(
+    spiderId: WebScraperId,
+    params?: WebScraperParams,
+    options?: WebScraperOptions,
+  ): Promise<T>;
+  scrapeResponse<T = unknown>(
+    spiderId: WebScraperId,
+    params?: WebScraperParams,
+    options?: WebScraperOptions,
+  ): Promise<WebScraperResponse<T>>;
+}
+
+export declare const Client: typeof ThorDataClient;
+export declare function search<T = unknown>(
+  params?: SearchParams,
+  options?: ThorDataClientOptions & RequestOptions,
+): Promise<T>;
+export declare function searchJson<T = unknown>(
+  params?: SearchParams,
+  options?: ThorDataClientOptions & RequestOptions,
+): Promise<T>;
+export declare function rawSearch(
+  params?: Record<string, unknown>,
+  options?: ThorDataClientOptions & RequestOptions,
+): Promise<string>;