llm-search-tools 1.1.0

package/docs/flights.md

@@ -0,0 +1,71 @@
+# Flights Search Documentation
+
+The flights module allows you to search for flight prices and schedules using Google Flights. It uses headless browser automation to navigate the interface and extract flight data.
+
+## Usage
+
+```typescript
+import { searchFlights } from "llm-search-tools";
+
+// Simple string query
+const results = await searchFlights("flights from JFK to LHR");
+
+// Structured query
+const results = await searchFlights({
+    from: "SFO",
+    to: "HND",
+    departureDate: "2025-05-01",
+    returnDate: "2025-05-15",
+});
+
+results.flights.forEach((flight) => {
+    console.log(`${flight.airline}: ${flight.price} (${flight.duration})`);
+});
+```
+
+## API Reference
+
+### `searchFlights(query, options?)`
+
+- **query**: `string | FlightSearchOptions` - Either a natural language string or an options object
+- **options**: `FlightSearchOptions` - Additional options (if query is a string)
+
+### Options (`FlightSearchOptions`)
+
+| Option          | Type                    | Description                                     |
+| --------------- | ----------------------- | ----------------------------------------------- |
+| `from`          | `string`                | Origin airport code or city                     |
+| `to`            | `string`                | Destination airport code or city                |
+| `departureDate` | `string`                | Departure date (YYYY-MM-DD or natural language) |
+| `returnDate`    | `string`                | Return date (YYYY-MM-DD or natural language)    |
+| `limit`         | `number`                | Maximum results (default: 10)                   |
+| `proxy`         | `string \| ProxyConfig` | Proxy configuration                             |
+
+## Output Structure
+
+Returns a `Promise<FlightResult>`:
+
+```typescript
+interface FlightResult {
+    flights: Flight[];
+    url: string; // Google Flights URL
+    source: string; // 'google-flights'
+}
+
+interface Flight {
+    airline: string;
+    departureTime: string;
+    arrivalTime: string;
+    duration: string;
+    price: string;
+    stops: string;
+    origin?: string; // Inferred if available
+    destination?: string; // Inferred if available
+}
+```
+
+## Limitations
+
+- Google Flights is a complex, dynamic application. Selectors are heavily obfuscated and change frequently.
+- This module uses heuristics to extract data (e.g., regex for times and prices).
+- Performance depends on network speed and proxy quality, as it loads a full React application.

package/docs/hackernews.md

@@ -0,0 +1,121 @@
+# HackerNews Module 💻
+
+Get the latest tech news and discussions from Hacker News.
+
+## Functions
+
+### getTopStories(limit?: number)
+
+Get the current top stories.
+
+```typescript
+import { getTopStories } from 'llm-search-tools';
+
+const stories = await getTopStories(10);
+```
+
+### getNewStories(limit?: number)
+
+Get the newest stories.
+
+```typescript
+import { getNewStories } from 'llm-search-tools';
+
+const stories = await getNewStories(10);
+```
+
+### getBestStories(limit?: number)
+
+Get the best stories of all time.
+
+```typescript
+import { getBestStories } from 'llm-search-tools';
+
+const stories = await getBestStories(10);
+```
+
+### getAskStories(limit?: number)
+
+Get "Ask HN" posts.
+
+```typescript
+import { getAskStories } from 'llm-search-tools';
+
+const stories = await getAskStories(10);
+```
+
+### getShowStories(limit?: number)
+
+Get "Show HN" posts.
+
+```typescript
+import { getShowStories } from 'llm-search-tools';
+
+const stories = await getShowStories(10);
+```
+
+### getJobStories(limit?: number)
+
+Get job postings.
+
+```typescript
+import { getJobStories } from 'llm-search-tools';
+
+const stories = await getJobStories(10);
+```
+
+### getStoryById(id: number)
+
+Get a specific story by its ID.
+
+```typescript
+import { getStoryById } from 'llm-search-tools';
+
+const story = await getStoryById(123456);
+```
+
+## Result Format
+
+```typescript
+interface HackerNewsResult extends SearchResult {
+  points?: number;     // story points/score
+  author?: string;     // post author
+  comments?: number;   // number of comments
+  time?: Date;        // post timestamp
+}
+```
+
+## Error Handling
+
+All functions throw a `SearchError` on failure:
+
+```typescript
+try {
+  const stories = await getTopStories();
+} catch (err) {
+  if (err.code === 'HN_TOP_ERROR') {
+    console.error('failed to get top stories:', err.message);
+  }
+}
+```
+
+## Tips
+
+- Default limit is 10 stories for all functions
+- Stories are returned in descending order (highest score first)
+- Use `getStoryById()` to fetch full details of a story
+- Comments count might be null for very new stories
+- URLs point to HN discussion if no external URL exists
+- Job stories won't have points or comments
+- "Ask HN" and "Show HN" often have interesting discussions
+
+## Story Types
+
+Each function gets different types of content:
+
+- `getTopStories()` - Currently trending stories
+- `getNewStories()` - Most recent submissions
+- `getBestStories()` - Highest voted all time
+- `getAskStories()` - Questions and discussions
+- `getShowStories()` - Project showcases
+- `getJobStories()` - Job listings

package/docs/media.md

@@ -0,0 +1,87 @@
+# Media Search Module 🎬
+
+The media search module provides unified access to metadata from **TMDB**, **TheTVDB**, and **AniDB** without requiring API keys. It uses a smart fallback strategy to find the best results for Movies, TV Shows, and Anime.
+
+## Functions
+
+### searchMedia(query: string, options?: MediaSearchOptions)
+
+Main function that orchestrates the search based on the requested media type.
+
+```typescript
+import { searchMedia } from "llm-search-tools";
+
+// General search (defaults to TMDB -> TheTVDB)
+const results = await searchMedia("Breaking Bad");
+
+// Specific Anime search (AniDB -> TMDB)
+const animeResults = await searchMedia("Neon Genesis Evangelion", {
+    type: "anime",
+});
+```
+
+## Strategies
+
+The module uses different strategies based on the `type` option:
+
+1.  **Anime** (`type: "anime"`)
+    - **Primary**: AniDB (Best for anime metadata)
+    - **Fallback**: TMDB (filtered for Animation/TV)
+
+2.  **TV** (`type: "tv"`)
+    - **Primary**: TMDB (Fast, good coverage)
+    - **Fallback**: TheTVDB (Specialized TV database)
+
+3.  **Movie / General** (Default)
+    - **Primary**: TMDB
+    - **Fallback**: TheTVDB (Only if not explicitly searching for movies)
+
+## Options
+
+```typescript
+interface MediaSearchOptions extends ScraperOptions {
+    type?: "movie" | "tv" | "anime";
+
+    // Inherited from ScraperOptions
+    limit?: number;
+    proxy?: ProxyConfig | string;
+    forcePuppeteer?: boolean; // Useful if getting blocked
+}
+```
+
+## Result Format
+
+Returns a consistent `MediaResult` object regardless of the source.
+
+```typescript
+interface MediaResult {
+    title: string;
+    url: string;
+    description?: string;
+    rating?: string; // e.g. "85%" or "8.5"
+    releaseDate?: string;
+    posterUrl?: string;
+    genres?: string[];
+    cast?: string[];
+
+    // Source identification
+    source: "tmdb" | "thetvdb" | "anidb";
+    mediaType: "movie" | "tv" | "anime";
+
+    // Availability (if scraped)
+    watchProviders?: {
+        name: string;
+        type: "stream" | "rent" | "buy";
+    }[];
+}
+```
+
+## Anti-Bot Features
+
+- **AniDB**: Enforces strict rate limiting (2.5s between requests) and always uses a stealth browser to avoid IP bans.
+- **TMDB/TheTVDB**: Starts with lightweight HTTP requests and automatically falls back to a stealth Puppeteer browser if bot protection is detected.
+
+## Tips
+
+- **Anime**: Always specify `{ type: "anime" }` for anime queries. AniDB has very strict matching but high-quality data.
+- **Rate Limiting**: If you are making many requests, add delays between them. The module handles internal rate limits for specific providers, but aggressive usage might still trigger captchas.

package/docs/news.md

@@ -0,0 +1,75 @@
+# News Module 📰
+
+The news module provides specialized news search capabilities using Google News and DuckDuckGo News.
+
+## Functions
+
+### searchNews(query: string, options?: ScraperOptions)
+
+Main news search function that orchestrates multiple providers for resilience:
+
+1. **Google News** (Primary source, rich metadata)
+2. **DuckDuckGo News** (Fallback source)
+
+```typescript
+import { searchNews } from "llm-search-tools";
+
+const results = await searchNews("technology trends", {
+    limit: 10,
+    timeout: 10000,
+});
+```
+
+### searchGoogleNews(query: string, options?: ScraperOptions)
+
+Search using Google News specifically. Uses the `google-news-scraper` library.
+
+```typescript
+import { searchGoogleNews } from "llm-search-tools";
+
+const results = await searchGoogleNews("AI developments");
+```
+
+## Options
+
+```typescript
+interface ScraperOptions {
+    limit?: number;        // max number of results (default: 10)
+    timeout?: number;      // request timeout in ms (default: 10000)
+    safeSearch?: boolean;  // enable safe search (default: true)
+
+    // Proxy configuration
+    proxy?: string | ProxyConfig;
+}
+```
+
+## Result Format
+
+The module returns `NewsResult` objects which extend the standard `SearchResult`:
+
+```typescript
+interface NewsResult extends SearchResult {
+    source: "google-news" | "duckduckgo-news";
+    sourceName?: string;   // The specific publisher (e.g., "The Verge", "CNN")
+    publishedAt?: string;  // Publication time string (e.g., "2 hours ago")
+    imageUrl?: string;     // URL to the article's thumbnail image
+}
+```
+
+## Caching
+
+The Google News provider implements in-memory caching (TTL: 30 minutes) to prevent rate limiting and improve performance for repeated queries.
+
+## Error Handling
+
+Like the search module, `searchNews` aggregates errors and only throws `ALL_NEWS_ENGINES_FAILED` if all providers fail.
+
+```typescript
+try {
+    const results = await searchNews("query");
+} catch (err) {
+    if (err.code === "ALL_NEWS_ENGINES_FAILED") {
+        console.error("News search failed:", err.errors);
+    }
+}
+```

package/docs/parser.md

@@ -0,0 +1,197 @@
+# Parser Module Documentation
+
+The parser module provides a unified interface for parsing various file types into text and structured data. It now supports both file paths and raw buffers as input.
+
+## Supported File Types
+
+- PDF documents (`.pdf`)
+- Word documents (`.docx`) via Mammoth
+- CSV files (`.csv`)
+- Images (`.png`, `.jpg`, `.jpeg`, `.bmp`, `.gif`) via OCR
+- Plain text files (`.txt`)
+- XML documents (`.xml`)
+- JSON files (`.json`)
+
+## Installation
+
+The module requires several dependencies:
+
+```bash
+npm install pdf-parse mammoth csv-parse tesseract.js fast-xml-parser
+```
+
+## Usage
+
+### Basic Usage
+
+```typescript
+import { parse } from "llm-search-tools";
+
+// parse a file by path (ez mode)
+const result = await parse("path/to/file.pdf");
+console.log(result.text);
+
+// parse a buffer with known filename (recommended for buffers)
+const buffer = readFileSync("path/to/file.docx");
+const result = await parse(buffer, {}, "file.docx");
+console.log(result.text);
+
+// parse a buffer without filename (we'll try our best to detect type)
+const buffer = someBuffer;
+const result = await parse(buffer);
+console.log(result.text);
+```
+
+### Type Detection for Buffers
+
+When parsing buffers, the parser attempts to detect the file type in several ways:
+
+1. Using the provided filename hint (most reliable)
+2. Checking file magic numbers for binary formats
+3. Attempting JSON parsing for potential JSON data
+4. Looking for CSV patterns
+5. Falling back to plain text if nothing else matches
+
+### With Options
+
+```typescript
+// Parse CSV with custom options
+const csvResult = await parse("data.csv", {
+    csv: {
+        delimiter: ";",
+        columns: true,
+    },
+});
+
+// OCR with different language
+const imageResult = await parse("image.png", {
+    language: "spa", // Spanish
+});
+```
+
+## Return Type
+
+The parser returns a `ParseResult` object:
+
+```typescript
+interface ParseResult {
+    type:
+        | "pdf"
+        | "docx"
+        | "csv"
+        | "image"
+        | "text"
+        | "xml"
+        | "json"
+        | "unknown";
+    text: string; // extracted text content
+    metadata?: Record<string, unknown>; // file metadata
+    data?: unknown; // structured data if we got it (xml/json/csv mostly)
+}
+```
+
+## File Type Specific Features
+
+### PDF Files
+
+- Extracts text content
+- Provides metadata (page count, PDF info, version)
+- Preserves document structure
+
+### DOCX Files
+
+- Extracts text content
+- Enhanced document handling via Mammoth
+- Supports both HTML and raw text extraction
+- Handles document formatting and structure
+- Automatically cleans and preserves formatting
+
+### CSV Files
+
+- Extracts raw text
+- Provides structured data array
+- Supports custom delimiters
+- Optional column headers
+
+### Images (OCR)
+
+- Extracts text via Tesseract OCR
+- Supports multiple languages
+- Returns confidence scores
+- Handles common image formats
+
+## Error Handling
+
+```typescript
+try {
+    const result = await parse("file.pdf");
+} catch (error) {
+    if (error.code === "PDF_PARSE_ERROR") {
+        // Handle PDF-specific error
+    } else if (error.code === "DOCX_PARSE_ERROR") {
+        // Handle DOCX-specific error
+    }
+    // Generic error handling
+    console.error(error.message);
+}
+```
+
+## Supported Error Codes
+
+- `PDF_PARSE_ERROR`: pdf parsing failed (ugh these pdfs i swear)
+- `DOCX_PARSE_ERROR`: docx parsing failed (word docs are the worst)
+- `CSV_PARSE_ERROR`: csv parsing went sideways
+- `IMAGE_PARSE_ERROR`: ocr failed (probably bad image quality or smth)
+- `TEXT_PARSE_ERROR`: somehow failed to parse plain text (how even?)
+- `XML_PARSE_ERROR`: xml parsing went wrong (invalid format probs)
+- `JSON_PARSE_ERROR`: json parsing died (check your brackets)
+- `PARSE_ERROR`: generic error when everything else fails
+
+## Language Support
+
+For OCR (image parsing), the following languages are supported:
+
+- English (default, 'eng')
+- Spanish ('spa')
+- French ('fra')
+- German ('deu')
+- And [many more](https://tesseract-ocr.github.io/tessdoc/Data-Files#data-files-for-version-400-november-29-2016)
+
+## Performance Considerations
+
+- Image OCR is CPU-intensive and may take longer
+- Large PDFs may require more memory
+- Consider using streams for large files
+- CSV parsing is typically very fast
+
+## Examples
+
+### Parse PDF and Extract Text
+
+```typescript
+const result = await parse("document.pdf");
+console.log(`Pages: ${result.metadata.pages}`);
+console.log(`Text: ${result.text}`);
+```
+
+### Parse CSV with Custom Options
+
+```typescript
+const result = await parse("data.csv", {
+    csv: {
+        delimiter: ";",
+        columns: true,
+    },
+});
+console.log(`Rows: ${result.metadata.rowCount}`);
+console.log(`Data:`, result.data);
+```
+
+### OCR Image in Different Language
+
+```typescript
+const result = await parse("chinese-text.png", {
+    language: "chi_sim",
+});
+console.log(`Extracted Text: ${result.text}`);
+```