@happyvertical/documents 0.74.9

package/AGENT.md

@@ -0,0 +1,32 @@
+# @happyvertical/documents
+
+<!-- BEGIN AGENT:GENERATED -->
+## Purpose
+Multi-part document processing with support for PDF, HTML, and Markdown
+
+## Package Map
+- Package: `@happyvertical/documents`
+- Hierarchy path: `@happyvertical/sdk > packages > documents`
+- Workspace position: `8 of 30` local packages
+- Internal dependencies: `@happyvertical/files`, `@happyvertical/utils`
+- Internal dependents: none
+- Knowledge graph files: `AGENT.md`, `metadata.json`, `ecosystem-manifest.json`
+
+## Build & Test
+```bash
+pnpm --filter @happyvertical/documents build
+pnpm --filter @happyvertical/documents test
+```
+
+## Agent Correction Loops
+- If module resolution or export errors mention a workspace dependency, build the dependency first (`pnpm --filter @happyvertical/files build`, `pnpm --filter @happyvertical/utils build`) and then rerun `pnpm --filter @happyvertical/documents build`.
+- If a change only affects runtime behavior, rerun `pnpm --filter @happyvertical/documents test` after rebuilding the package to confirm the failure is local.
+- If failures span multiple packages or Turborepo ordering looks wrong, run `pnpm build` and `pnpm typecheck` from the repo root before retrying package-scoped commands.
+
+## Ecosystem Relationships
+- Provides: Multi-part document processing with support for PDF, HTML, and Markdown
+- Implements: none
+- Requires: @happyvertical/files, @happyvertical/utils, @happyvertical/ocr, @happyvertical/pdf, @happyvertical/spider, uuid
+- Stability: stable (Primary package surface is described as implemented and production-oriented.)
+<!-- END AGENT:GENERATED -->
+

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright <2025> <Happy Vertical Corporation>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,145 @@
+# @happyvertical/documents
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Document processing with hierarchical structure. Currently supports PDF documents with text extraction, automatic document management system detection (WordPress Download Manager, CivicWeb, DocuShare), and file caching. Uses `@happyvertical/spider` for web page analysis and `@happyvertical/pdf` for PDF text extraction.
+
+## Installation
+
+Install from the public npm registry:
+
+```bash
+npm install @happyvertical/documents
+```
+
+Anonymous installs also require the package's external `@happyvertical/ocr`, `@happyvertical/pdf`, and `@happyvertical/spider` dependencies to be available on public npm.
+
+## Quick Start
+
+```typescript
+import { fetchDocument } from '@happyvertical/documents';
+
+// Process a local PDF
+const doc = await fetchDocument('file:///path/to/report.pdf');
+
+for (const part of doc.parts) {
+  console.log(part.title);
+  console.log(part.content);
+}
+
+// Fetch a remote PDF (auto-detected from URL extension)
+const remote = await fetchDocument('https://example.com/report.pdf');
+console.log(remote.parts[0].content);
+```
+
+## Usage
+
+### Document Management System Detection
+
+When fetching web URLs, the package uses `@happyvertical/spider` to detect document management systems and extract direct PDF links:
+
+```typescript
+// WordPress Download Manager URL — spider detects the PDF link automatically
+const doc = await fetchDocument(
+  'https://example.com/download/meeting-minutes/',
+  { scraper: 'basic', spider: 'dom' }
+);
+```
+
+### Override MIME Type
+
+```typescript
+const doc = await fetchDocument('https://example.com/download?id=123', {
+  type: 'application/pdf',
+});
+```
+
+### Cache Control
+
+```typescript
+const doc = await fetchDocument('https://example.com/report.pdf', {
+  cacheDir: './my-cache',
+  cache: true,
+  cacheExpiry: 600_000, // 10 minutes
+});
+```
+
+## API Reference
+
+### `fetchDocument(url, options?)`
+
+Main factory function. Detects document format, selects the appropriate processor, and returns structured content.
+
+- **url** `string` — Document URL or file path (`file://`, `http://`, `https://`)
+- **options** `FetchDocumentOptions` — See below
+- **Returns** `Promise<Document>`
+- **Throws** if no processor is available for the detected MIME type
+
+### `FetchDocumentOptions`
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `type` | `string` | auto-detected | Override MIME type detection |
+| `extractImages` | `boolean` | `true` | Extract images from document (stub — currently returns `[]`) |
+| `runOcr` | `boolean` | `true` for PDFs | Run OCR on extracted images (stub) |
+| `cacheDir` | `string` | OS temp dir | Directory for caching downloaded files |
+| `cache` | `boolean` | `true` | Enable/disable spider fetch caching |
+| `cacheExpiry` | `number` | `300000` | Cache expiry in milliseconds |
+| `scraper` | `'basic' \| 'crawlee'` | `'basic'` | Scraper type for content extraction |
+| `spider` | `'simple' \| 'dom' \| 'crawlee'` | `'dom'` | Spider adapter for fetching web pages |
+| `headers` | `Record<string, string>` | — | Custom HTTP headers for spider requests |
+| `timeout` | `number` | `30000` | Request timeout in milliseconds |
+| `maxDuration` | `number` | — | Max scraping time in milliseconds |
+| `maxInteractions` | `number` | — | Max interactions for advanced scrapers |
+
+### `Document` (class)
+
+Base document handler. Manages downloading, caching, and local file path resolution. Used internally by processors; can also be used directly via `Document.create(url, options)`.
+
+### `PDFProcessor`
+
+Implements `DocumentProcessor`. Extracts text from PDF files, validates PDF headers (detects HTML cache poisoning), and caches processed results.
+
+### `getTitleFromUrl(url, defaultTitle?)`
+
+Extracts a human-readable title from a URL by parsing the filename, removing extensions, and decoding URL-encoded characters.
+
+### Types
+
+```typescript
+interface Document {
+  url: string;
+  type: string;
+  parts: DocumentPart[];
+  metadata?: Record<string, any>;
+}
+
+interface DocumentPart {
+  id: string;
+  title: string;
+  content: string;
+  type: 'text' | 'html' | 'markdown';
+  images?: DocumentImage[];
+  metadata?: Record<string, any>;
+  parts?: DocumentPart[];
+}
+
+interface DocumentImage {
+  id: string;
+  url: string;
+  localPath?: string;
+  altText?: string;
+  ocrText?: string;
+  position?: number;
+  metadata?: { width?: number; height?: number; format?: string };
+}
+
+interface DocumentProcessor {
+  process(url: string, options?: FetchDocumentOptions): Promise<Document>;
+  supports(type: string): boolean;
+}
+```
+
+## License
+
+MIT

package/dist/cli/claude-context.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+export {};
+//# sourceMappingURL=claude-context.d.ts.map

package/dist/cli/claude-context.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"claude-context.d.ts","sourceRoot":"","sources":["../../src/cli/claude-context.ts"],"names":[],"mappings":""}

package/dist/cli/claude-context.js

@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+import { existsSync, mkdirSync, copyFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+const Dirname = dirname(fileURLToPath(import.meta.url));
+const pkgRoot = join(Dirname, "../..");
+const targetDir = join(process.cwd(), ".claude");
+if (!existsSync(targetDir)) {
+  mkdirSync(targetDir, { recursive: true });
+}
+const pkgName = "documents";
+const agentMdSrc = existsSync(join(pkgRoot, "AGENT.md")) ? join(pkgRoot, "AGENT.md") : join(pkgRoot, "CLAUDE.md");
+const metaSrc = existsSync(join(pkgRoot, "metadata.json")) ? join(pkgRoot, "metadata.json") : join(pkgRoot, ".claude-meta.json");
+if (existsSync(agentMdSrc)) {
+  copyFileSync(agentMdSrc, join(targetDir, `have-${pkgName}.md`));
+}
+if (existsSync(metaSrc)) {
+  copyFileSync(metaSrc, join(targetDir, `have-${pkgName}.meta.json`));
+}
+console.log(`✓ Installed @happyvertical/${pkgName} context to .claude/`);
+//# sourceMappingURL=claude-context.js.map

package/dist/cli/claude-context.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"claude-context.js","sources":["../../src/cli/claude-context.ts"],"sourcesContent":["#!/usr/bin/env node\n/**\n * CLI script to install agent context for @happyvertical/documents\n * Run the published context installer binary for this package.\n */\nimport { copyFileSync, existsSync, mkdirSync } from 'node:fs';\nimport { dirname, join } from 'node:path';\nimport { fileURLToPath } from 'node:url';\n\nconst Dirname = dirname(fileURLToPath(import.meta.url));\nconst pkgRoot = join(Dirname, '../..');\nconst targetDir = join(process.cwd(), '.claude');\n\nif (!existsSync(targetDir)) {\n  mkdirSync(targetDir, { recursive: true });\n}\n\nconst pkgName = 'documents';\nconst agentMdSrc = existsSync(join(pkgRoot, 'AGENT.md'))\n  ? join(pkgRoot, 'AGENT.md')\n  : join(pkgRoot, 'CLAUDE.md');\nconst metaSrc = existsSync(join(pkgRoot, 'metadata.json'))\n  ? join(pkgRoot, 'metadata.json')\n  : join(pkgRoot, '.claude-meta.json');\n\nif (existsSync(agentMdSrc)) {\n  copyFileSync(agentMdSrc, join(targetDir, `have-${pkgName}.md`));\n}\n\nif (existsSync(metaSrc)) {\n  copyFileSync(metaSrc, join(targetDir, `have-${pkgName}.meta.json`));\n}\n\nconsole.log(`✓ Installed @happyvertical/${pkgName} context to .claude/`);\n"],"names":[],"mappings":";;;;AASA,MAAM,UAAU,QAAQ,cAAc,YAAY,GAAG,CAAC;AACtD,MAAM,UAAU,KAAK,SAAS,OAAO;AACrC,MAAM,YAAY,KAAK,QAAQ,IAAA,GAAO,SAAS;AAE/C,IAAI,CAAC,WAAW,SAAS,GAAG;AAC1B,YAAU,WAAW,EAAE,WAAW,KAAA,CAAM;AAC1C;AAEA,MAAM,UAAU;AAChB,MAAM,aAAa,WAAW,KAAK,SAAS,UAAU,CAAC,IACnD,KAAK,SAAS,UAAU,IACxB,KAAK,SAAS,WAAW;AAC7B,MAAM,UAAU,WAAW,KAAK,SAAS,eAAe,CAAC,IACrD,KAAK,SAAS,eAAe,IAC7B,KAAK,SAAS,mBAAmB;AAErC,IAAI,WAAW,UAAU,GAAG;AAC1B,eAAa,YAAY,KAAK,WAAW,QAAQ,OAAO,KAAK,CAAC;AAChE;AAEA,IAAI,WAAW,OAAO,GAAG;AACvB,eAAa,SAAS,KAAK,WAAW,QAAQ,OAAO,YAAY,CAAC;AACpE;AAEA,QAAQ,IAAI,8BAA8B,OAAO,sBAAsB;"}

package/dist/document.d.ts

@@ -0,0 +1,88 @@
+import { URL } from 'node:url';
+import { DocumentPart, Document as DocumentType, FetchDocumentOptions } from './types';
+/**
+ * Base document handler with multi-part support
+ *
+ * Provides functionality for downloading, caching, and structuring documents
+ * into hierarchical parts. Specific format processing (PDF, HTML, Markdown)
+ * is handled by specialized processors.
+ */
+export declare class Document {
+    /**
+     * Flag indicating if document is from a remote source
+     */
+    protected isRemote: boolean;
+    /**
+     * Configuration options
+     */
+    protected options: FetchDocumentOptions;
+    /**
+     * Local file path where document is stored
+     */
+    private _localPath;
+    /**
+     * Directory used for caching files
+     */
+    private _cacheDir;
+    /**
+     * Document URL
+     */
+    url: URL;
+    /**
+     * Document MIME type
+     */
+    type: string;
+    /**
+     * Document parts (hierarchical structure)
+     */
+    parts: DocumentPart[];
+    /**
+     * Document-level metadata
+     */
+    metadata: Record<string, any>;
+    /**
+     * Get the local file path where document is stored
+     */
+    get localPath(): string;
+    /**
+     * Get the directory used for caching files
+     */
+    get cacheDir(): string;
+    /**
+     * Creates a new Document instance
+     *
+     * @param url - Document URL or file path
+     * @param options - Document configuration options
+     */
+    constructor(url: string, options?: FetchDocumentOptions);
+    /**
+     * Creates and initializes a Document instance
+     *
+     * Downloads remote files and prepares the document for processing.
+     *
+     * @param url - Document URL or file path
+     * @param options - Document configuration options
+     * @returns Promise resolving to the initialized Document
+     */
+    static create(url: string, options?: FetchDocumentOptions): Promise<Document>;
+    /**
+     * Initializes the document, downloading it if it's remote
+     *
+     * @returns Promise that resolves when initialization is complete
+     */
+    initialize(): Promise<void>;
+    /**
+     * Checks if the document is a text-based file that can be read directly
+     *
+     * @returns Boolean indicating if the file is text-based
+     */
+    isTextFile(): boolean;
+    /**
+     * Converts the document to the standard Document interface
+     *
+     * @returns Document object with URL, type, parts, and metadata
+     */
+    toDocument(): DocumentType;
+}
+export default Document;
+//# sourceMappingURL=document.d.ts.map

package/dist/document.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"document.d.ts","sourceRoot":"","sources":["../src/document.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAG/B,OAAO,KAAK,EACV,YAAY,EACZ,QAAQ,IAAI,YAAY,EACxB,oBAAoB,EACrB,MAAM,SAAS,CAAC;AAEjB;;;;;;GAMG;AACH,qBAAa,QAAQ;IACnB;;OAEG;IACH,SAAS,CAAC,QAAQ,UAAS;IAE3B;;OAEG;IACH,SAAS,CAAC,OAAO,EAAE,oBAAoB,CAAC;IAExC;;OAEG;IACH,OAAO,CAAC,UAAU,CAAM;IAExB;;OAEG;IACH,OAAO,CAAC,SAAS,CAAM;IAEvB;;OAEG;IACI,GAAG,EAAE,GAAG,CAAC;IAEhB;;OAEG;IACI,IAAI,EAAE,MAAM,CAAC;IAEpB;;OAEG;IACI,KAAK,EAAE,YAAY,EAAE,CAAM;IAElC;;OAEG;IACI,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAM;IAE1C;;OAEG;IACH,IAAW,SAAS,IAAI,MAAM,CAE7B;IAED;;OAEG;IACH,IAAW,QAAQ,IAAI,MAAM,CAE5B;IAED;;;;;OAKG;gBACS,GAAG,EAAE,MAAM,EAAE,OAAO,GAAE,oBAAyB;IA+C3D;;;;;;;;OAQG;WACU,MAAM,CACjB,GAAG,EAAE,MAAM,EACX,OAAO,GAAE,oBAAyB,GACjC,OAAO,CAAC,QAAQ,CAAC;IAMpB;;;;OAIG;IACG,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IASjC;;;;OAIG;IACI,UAAU,IAAI,OAAO;IAwB5B;;;;OAIG;IACI,UAAU,IAAI,YAAY;CAQlC;AAED,eAAe,QAAQ,CAAC"}

package/dist/factory.d.ts

@@ -0,0 +1,40 @@
+import { Document, FetchDocumentOptions } from './types';
+/**
+ * Fetch a document from a URL with automatic format detection
+ *
+ * This factory function:
+ * 1. Detects the document format (PDF, HTML, Markdown, etc.)
+ * 2. Selects the appropriate processor
+ * 3. Processes the document into structured parts
+ * 4. Returns a Document object with hierarchical content
+ *
+ * @param url - Document URL or file path (file://, http://, https://)
+ * @param options - Fetch and processing options
+ * @returns Promise resolving to structured Document
+ *
+ * @example
+ * ```typescript
+ * // Fetch a PDF with image extraction and OCR
+ * const doc = await fetchDocument('https://example.com/report.pdf', {
+ *   extractImages: true,
+ *   runOcr: true
+ * });
+ *
+ * // Access document parts
+ * for (const part of doc.parts) {
+ *   console.log(part.title);
+ *   console.log(part.content);
+ *
+ *   // Check for images
+ *   if (part.images) {
+ *     for (const image of part.images) {
+ *       console.log(image.url);
+ *       console.log(image.ocrText); // Text extracted via OCR
+ *     }
+ *   }
+ * }
+ * ```
+ */
+export declare function fetchDocument(url: string, options?: FetchDocumentOptions): Promise<Document>;
+export default fetchDocument;
+//# sourceMappingURL=factory.d.ts.map

package/dist/factory.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"factory.d.ts","sourceRoot":"","sources":["../src/factory.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,oBAAoB,EAAE,MAAM,SAAS,CAAC;AAO9D;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAmCG;AACH,wBAAsB,aAAa,CACjC,GAAG,EAAE,MAAM,EACX,OAAO,GAAE,oBAAyB,GACjC,OAAO,CAAC,QAAQ,CAAC,CAyEnB;AAED,eAAe,aAAa,CAAC"}

package/dist/index.d.ts

@@ -0,0 +1,29 @@
+/**
+ * @happyvertical/documents - Document processing with multi-part structure
+ *
+ * Provides document processing for PDFs with support for:
+ * - Hierarchical document parts
+ * - Automatic format detection from URL or MIME type
+ * - Document management system detection (WordPress, CivicWeb, DocuShare)
+ * - File caching for performance
+ *
+ * @example
+ * ```typescript
+ * import { fetchDocument } from '@happyvertical/documents';
+ *
+ * const doc = await fetchDocument('https://example.com/report.pdf');
+ *
+ * for (const part of doc.parts) {
+ *   console.log(part.title);
+ *   console.log(part.content);
+ * }
+ * ```
+ */
+export { Document } from './document';
+export { fetchDocument } from './factory';
+export { PDFProcessor } from './processors/pdf';
+export type { Document as DocumentType, DocumentImage, DocumentPart, DocumentProcessor, FetchDocumentOptions, } from './types';
+export { getTitleFromUrl } from './utils';
+/** @internal */
+export declare const PACKAGE_VERSION_INITIALIZED = true;
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAGH,OAAO,EAAE,QAAQ,EAAE,MAAM,YAAY,CAAC;AAEtC,OAAO,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAG1C,OAAO,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAEhD,YAAY,EACV,QAAQ,IAAI,YAAY,EACxB,aAAa,EACb,YAAY,EACZ,iBAAiB,EACjB,oBAAoB,GACrB,MAAM,SAAS,CAAC;AAEjB,OAAO,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAE1C,gBAAgB;AAChB,eAAO,MAAM,2BAA2B,OAAO,CAAC"}

package/dist/index.js

@@ -0,0 +1,320 @@
+import os from "node:os";
+import path from "node:path";
+import { URL as URL$1 } from "node:url";
+import { getMimeType, downloadFileWithCache, getCached, setCached } from "@happyvertical/files";
+import { makeSlug } from "@happyvertical/utils";
+import { scrapeDocument } from "@happyvertical/spider";
+import { promises } from "node:fs";
+import { getPDFReader } from "@happyvertical/pdf";
+import { v4 } from "uuid";
+class Document {
+  /**
+   * Flag indicating if document is from a remote source
+   */
+  isRemote = false;
+  /**
+   * Configuration options
+   */
+  options;
+  /**
+   * Local file path where document is stored
+   */
+  _localPath = "";
+  /**
+   * Directory used for caching files
+   */
+  _cacheDir = "";
+  /**
+   * Document URL
+   */
+  url;
+  /**
+   * Document MIME type
+   */
+  type;
+  /**
+   * Document parts (hierarchical structure)
+   */
+  parts = [];
+  /**
+   * Document-level metadata
+   */
+  metadata = {};
+  /**
+   * Get the local file path where document is stored
+   */
+  get localPath() {
+    return this._localPath;
+  }
+  /**
+   * Get the directory used for caching files
+   */
+  get cacheDir() {
+    return this._cacheDir;
+  }
+  /**
+   * Creates a new Document instance
+   *
+   * @param url - Document URL or file path
+   * @param options - Document configuration options
+   */
+  constructor(url, options = {}) {
+    this.url = new URL$1(url);
+    this.options = options;
+    this.type = options.type || getMimeType(this.url.toString()) || "text/plain";
+    this._cacheDir = options.cacheDir || path.resolve(os.tmpdir(), ".cache", "have-sdk", "documents");
+    if (this.url.protocol.startsWith("file")) {
+      this._localPath = decodeURIComponent(this.url.pathname);
+      this.isRemote = false;
+    } else if (this.url.protocol.startsWith("http")) {
+      let pathname = this.url.pathname;
+      if (pathname.endsWith("/")) {
+        pathname = pathname.slice(0, -1);
+      }
+      if (!pathname.match(/\.[a-z0-9]+$/i)) {
+        if (this.type === "application/pdf" || options.type === "application/pdf") {
+          pathname += ".pdf";
+        }
+      }
+      this._localPath = path.join(
+        this._cacheDir,
+        makeSlug(this.url.hostname),
+        pathname
+      );
+      this.isRemote = true;
+    }
+  }
+  /**
+   * Creates and initializes a Document instance
+   *
+   * Downloads remote files and prepares the document for processing.
+   *
+   * @param url - Document URL or file path
+   * @param options - Document configuration options
+   * @returns Promise resolving to the initialized Document
+   */
+  static async create(url, options = {}) {
+    const document = new Document(url, options);
+    await document.initialize();
+    return document;
+  }
+  /**
+   * Initializes the document, downloading it if it's remote
+   *
+   * @returns Promise that resolves when initialization is complete
+   */
+  async initialize() {
+    if (this.isRemote) {
+      if (!this.url) {
+        throw new Error("Cannot initialize remote document: URL is required");
+      }
+      await downloadFileWithCache(this.url.toString(), this._localPath);
+    }
+  }
+  /**
+   * Checks if the document is a text-based file that can be read directly
+   *
+   * @returns Boolean indicating if the file is text-based
+   */
+  isTextFile() {
+    if (!this.type) return false;
+    return this.type.startsWith("text/") || this.type === "application/json" || this.type === "application/xml" || this.type === "application/javascript" || this.type === "application/typescript" || [
+      ".txt",
+      ".md",
+      ".json",
+      ".xml",
+      ".html",
+      ".css",
+      ".js",
+      ".ts",
+      ".yaml",
+      ".yml"
+    ].some((ext) => this.localPath.toLowerCase().endsWith(ext));
+  }
+  /**
+   * Converts the document to the standard Document interface
+   *
+   * @returns Document object with URL, type, parts, and metadata
+   */
+  toDocument() {
+    return {
+      url: this.url.toString(),
+      type: this.type,
+      parts: this.parts,
+      metadata: this.metadata
+    };
+  }
+}
+function getTitleFromUrl(url, defaultTitle = "Document") {
+  try {
+    const urlObj = new URL(url);
+    const pathname = urlObj.pathname;
+    const filename = pathname.split("/").pop() || defaultTitle;
+    const decodedFilename = decodeURIComponent(filename);
+    return decodedFilename.replace(/\.(pdf|html?|md|txt)$/i, "").replace(/[-_]/g, " ").trim();
+  } catch {
+    return defaultTitle;
+  }
+}
+class PDFProcessor {
+  /**
+   * Check if this processor supports the given MIME type or extension.
+   * Accepts `'application/pdf'`, `'.pdf'`, or `'pdf'` (case-insensitive).
+   *
+   * @param type - MIME type or file extension to check
+   * @returns `true` if this processor can handle the given type
+   */
+  supports(type) {
+    return type === "application/pdf" || type.endsWith(".pdf") || type.toLowerCase() === "pdf";
+  }
+  /**
+   * Process a PDF document
+   *
+   * Extracts text and optionally images/OCR from the PDF, structuring
+   * it into hierarchical document parts.
+   *
+   * @param url - PDF URL or file path
+   * @param options - Processing options
+   * @returns Promise resolving to structured Document
+   */
+  async process(url, options = {}) {
+    const baseDoc = await Document.create(url, options);
+    const cacheKey = `${baseDoc.localPath}.processed_pdf`;
+    const cached = await getCached(cacheKey);
+    if (cached) {
+      try {
+        const parsed = JSON.parse(cached);
+        return {
+          url: baseDoc.url.toString(),
+          type: baseDoc.type,
+          parts: parsed.parts,
+          metadata: parsed.metadata || {}
+        };
+      } catch (error) {
+        console.warn("Cached PDF data corrupted, reprocessing", error);
+      }
+    }
+    const fileBuffer = await promises.readFile(baseDoc.localPath);
+    const header = fileBuffer.subarray(0, 5).toString("utf-8");
+    if (header !== "%PDF-") {
+      try {
+        await promises.unlink(baseDoc.localPath);
+      } catch (unlinkError) {
+        console.warn(
+          `Failed to delete poisoned cache file: ${baseDoc.localPath}`,
+          unlinkError
+        );
+      }
+      const content = fileBuffer.toString(
+        "utf-8",
+        0,
+        Math.min(1e3, fileBuffer.length)
+      );
+      if (content.includes("<!DOCTYPE html>") || content.includes("<html")) {
+        throw new Error(
+          `Downloaded file is HTML, not PDF. The server returned HTML content for ${url}. This commonly occurs with WordPress Download Manager URLs that return tracking pages. Expected PDF magic bytes (%PDF-) but got: ${header}. The poisoned cache file has been removed - please try again.`
+        );
+      } else {
+        throw new Error(
+          `Downloaded file is not a valid PDF. Expected %PDF- magic bytes but got: ${header}. The invalid cache file has been removed - please try again.`
+        );
+      }
+    }
+    const reader = await getPDFReader();
+    const extractedText = await reader.extractText(baseDoc.localPath);
+    const mainPart = {
+      id: v4(),
+      title: getTitleFromUrl(url, "PDF Document"),
+      content: extractedText || "",
+      type: "text",
+      metadata: {
+        source: "pdf",
+        filePath: baseDoc.localPath
+      }
+    };
+    if (options.extractImages === true) {
+      mainPart.images = await this.extractImages(
+        baseDoc.localPath,
+        options.runOcr !== false
+      );
+    }
+    const document = {
+      url: baseDoc.url.toString(),
+      type: baseDoc.type,
+      parts: [mainPart],
+      metadata: {
+        processor: "pdf",
+        extractedAt: (/* @__PURE__ */ new Date()).toISOString(),
+        hasImages: (mainPart.images?.length || 0) > 0
+      }
+    };
+    await setCached(cacheKey, JSON.stringify(document));
+    return document;
+  }
+  /**
+   * Extract images from PDF
+   *
+   * This is a placeholder for future image extraction functionality.
+   * Will use @happyvertical/pdf's image extraction capabilities when available.
+   *
+   * @param filePath - Local PDF file path
+   * @param runOcr - Whether to run OCR on extracted images
+   * @returns Promise resolving to array of DocumentImages
+   */
+  async extractImages(_filePath, _runOcr) {
+    return [];
+  }
+}
+const processors = [new PDFProcessor()];
+async function fetchDocument(url, options = {}) {
+  const isWebUrl = url.startsWith("http://") || url.startsWith("https://");
+  if (isWebUrl && !options.type) {
+    try {
+      const scraped = await scrapeDocument(url, {
+        scraper: options.scraper || "basic",
+        spider: options.spider || "dom",
+        cache: options.cache,
+        cacheExpiry: options.cacheExpiry,
+        headers: options.headers,
+        timeout: options.timeout,
+        maxDuration: options.maxDuration,
+        maxInteractions: options.maxInteractions
+      });
+      const hasDocLink = scraped.metadata.strategy === "wordpress-pdf-link" || scraped.metadata.strategy === "civicweb-pdf-link" || scraped.metadata.strategy === "docushare-pdf-link";
+      if (hasDocLink && scraped.metadata.isPdf && !scraped.metadata.complete) {
+        url = scraped.url;
+        options.type = "application/pdf";
+      }
+    } catch (error) {
+      console.warn(
+        `Spider detection failed for ${url}, falling back to direct download:`,
+        error
+      );
+    }
+  }
+  let type = options.type;
+  if (!type) {
+    const urlLower = url.toLowerCase();
+    if (urlLower.endsWith(".pdf") || urlLower.includes(".pdf?") || urlLower.includes(".pdf#")) {
+      type = "application/pdf";
+    } else {
+      type = getMimeType(url) || "";
+    }
+  }
+  const processor = processors.find((p) => p.supports(type));
+  if (!processor) {
+    throw new Error(
+      `No processor available for document type: ${type}. Supported types: PDF (.pdf, application/pdf)`
+    );
+  }
+  return processor.process(url, options);
+}
+const PACKAGE_VERSION_INITIALIZED = true;
+export {
+  Document,
+  PACKAGE_VERSION_INITIALIZED,
+  PDFProcessor,
+  fetchDocument,
+  getTitleFromUrl
+};
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sources":["../src/document.ts","../src/utils.ts","../src/processors/pdf.ts","../src/factory.ts","../src/index.ts"],"sourcesContent":["import os from 'node:os';\nimport path from 'node:path';\nimport { URL } from 'node:url';\nimport { downloadFileWithCache, getMimeType } from '@happyvertical/files';\nimport { makeSlug } from '@happyvertical/utils';\nimport type {\n  DocumentPart,\n  Document as DocumentType,\n  FetchDocumentOptions,\n} from './types';\n\n/**\n * Base document handler with multi-part support\n *\n * Provides functionality for downloading, caching, and structuring documents\n * into hierarchical parts. Specific format processing (PDF, HTML, Markdown)\n * is handled by specialized processors.\n */\nexport class Document {\n  /**\n   * Flag indicating if document is from a remote source\n   */\n  protected isRemote = false;\n\n  /**\n   * Configuration options\n   */\n  protected options: FetchDocumentOptions;\n\n  /**\n   * Local file path where document is stored\n   */\n  private _localPath = '';\n\n  /**\n   * Directory used for caching files\n   */\n  private _cacheDir = '';\n\n  /**\n   * Document URL\n   */\n  public url: URL;\n\n  /**\n   * Document MIME type\n   */\n  public type: string;\n\n  /**\n   * Document parts (hierarchical structure)\n   */\n  public parts: DocumentPart[] = [];\n\n  /**\n   * Document-level metadata\n   */\n  public metadata: Record<string, any> = {};\n\n  /**\n   * Get the local file path where document is stored\n   */\n  public get localPath(): string {\n    return this._localPath;\n  }\n\n  /**\n   * Get the directory used for caching files\n   */\n  public get cacheDir(): string {\n    return this._cacheDir;\n  }\n\n  /**\n   * Creates a new Document instance\n   *\n   * @param url - Document URL or file path\n   * @param options - Document configuration options\n   */\n  constructor(url: string, options: FetchDocumentOptions = {}) {\n    this.url = new URL(url);\n    this.options = options;\n    this.type =\n      options.type || getMimeType(this.url.toString()) || 'text/plain';\n\n    this._cacheDir =\n      options.cacheDir ||\n      path.resolve(os.tmpdir(), '.cache', 'have-sdk', 'documents');\n\n    if (this.url.protocol.startsWith('file')) {\n      // Decode URL-encoded characters in the pathname only (e.g., %20 -> space).\n      // Note: Query parameters and hash fragments are not decoded here.\n      this._localPath = decodeURIComponent(this.url.pathname);\n      this.isRemote = false;\n    } else if (this.url.protocol.startsWith('http')) {\n      // Generate cache path from URL pathname\n      // Query parameters (?) and fragments (#) are automatically excluded from url.pathname\n      let pathname = this.url.pathname;\n\n      // Remove trailing slash (directory-style URLs)\n      if (pathname.endsWith('/')) {\n        pathname = pathname.slice(0, -1);\n      }\n\n      // Add file extension if missing and we know the type\n      // This is crucial for URLs like /download/file/?wpdmdl=123 which have no extension\n      if (!pathname.match(/\\.[a-z0-9]+$/i)) {\n        // Add appropriate extension based on MIME type\n        if (\n          this.type === 'application/pdf' ||\n          options.type === 'application/pdf'\n        ) {\n          pathname += '.pdf';\n        }\n        // Future: Add other common extensions (html, json, etc.)\n      }\n\n      this._localPath = path.join(\n        this._cacheDir,\n        makeSlug(this.url.hostname),\n        pathname,\n      );\n      this.isRemote = true;\n    }\n  }\n\n  /**\n   * Creates and initializes a Document instance\n   *\n   * Downloads remote files and prepares the document for processing.\n   *\n   * @param url - Document URL or file path\n   * @param options - Document configuration options\n   * @returns Promise resolving to the initialized Document\n   */\n  static async create(\n    url: string,\n    options: FetchDocumentOptions = {},\n  ): Promise<Document> {\n    const document = new Document(url, options);\n    await document.initialize();\n    return document;\n  }\n\n  /**\n   * Initializes the document, downloading it if it's remote\n   *\n   * @returns Promise that resolves when initialization is complete\n   */\n  async initialize(): Promise<void> {\n    if (this.isRemote) {\n      if (!this.url) {\n        throw new Error('Cannot initialize remote document: URL is required');\n      }\n      await downloadFileWithCache(this.url.toString(), this._localPath);\n    }\n  }\n\n  /**\n   * Checks if the document is a text-based file that can be read directly\n   *\n   * @returns Boolean indicating if the file is text-based\n   */\n  public isTextFile(): boolean {\n    if (!this.type) return false;\n\n    return (\n      this.type.startsWith('text/') ||\n      this.type === 'application/json' ||\n      this.type === 'application/xml' ||\n      this.type === 'application/javascript' ||\n      this.type === 'application/typescript' ||\n      [\n        '.txt',\n        '.md',\n        '.json',\n        '.xml',\n        '.html',\n        '.css',\n        '.js',\n        '.ts',\n        '.yaml',\n        '.yml',\n      ].some((ext) => this.localPath.toLowerCase().endsWith(ext))\n    );\n  }\n\n  /**\n   * Converts the document to the standard Document interface\n   *\n   * @returns Document object with URL, type, parts, and metadata\n   */\n  public toDocument(): DocumentType {\n    return {\n      url: this.url.toString(),\n      type: this.type,\n      parts: this.parts,\n      metadata: this.metadata,\n    };\n  }\n}\n\nexport default Document;\n","/**\n * Utility functions for document processing\n */\n\n/**\n * Extract a human-readable title from a URL\n *\n * Takes a URL and extracts the filename from the pathname, then formats it\n * into a readable title by removing the extension and converting separators\n * to spaces. Also decodes URL-encoded characters like %20.\n *\n * @param url - URL string to extract title from\n * @param defaultTitle - Default title to use if extraction fails\n * @returns Formatted title string\n *\n * @example\n * ```typescript\n * getTitleFromUrl('file:///path/to/My%20Document.pdf')\n * // Returns: 'My Document'\n *\n * getTitleFromUrl('https://example.com/research_paper.pdf')\n * // Returns: 'research paper'\n * ```\n */\nexport function getTitleFromUrl(\n  url: string,\n  defaultTitle = 'Document',\n): string {\n  try {\n    const urlObj = new URL(url);\n    const pathname = urlObj.pathname;\n    const filename = pathname.split('/').pop() || defaultTitle;\n\n    // Decode URL-encoded characters (e.g., %20 -> space)\n    const decodedFilename = decodeURIComponent(filename);\n\n    // Remove extension and convert separators to spaces\n    return decodedFilename\n      .replace(/\\.(pdf|html?|md|txt)$/i, '')\n      .replace(/[-_]/g, ' ')\n      .trim();\n  } catch {\n    return defaultTitle;\n  }\n}\n","import { promises as fs } from 'node:fs';\nimport { getCached, setCached } from '@happyvertical/files';\nimport { getPDFReader } from '@happyvertical/pdf';\nimport { v4 as uuidv4 } from 'uuid';\nimport { Document as BaseDocument } from '../document';\nimport type {\n  Document,\n  DocumentImage,\n  DocumentPart,\n  DocumentProcessor,\n  FetchDocumentOptions,\n} from '../types';\nimport { getTitleFromUrl } from '../utils';\n\n/**\n * PDF Document Processor\n *\n * Handles PDF documents with support for:\n * - Text extraction from PDF content via `@happyvertical/pdf`\n * - PDF header validation (detects HTML cache poisoning from document management systems)\n * - Processed document caching via `@happyvertical/files`\n *\n * Image extraction and OCR are stubbed for future implementation.\n */\nexport class PDFProcessor implements DocumentProcessor {\n  /**\n   * Check if this processor supports the given MIME type or extension.\n   * Accepts `'application/pdf'`, `'.pdf'`, or `'pdf'` (case-insensitive).\n   *\n   * @param type - MIME type or file extension to check\n   * @returns `true` if this processor can handle the given type\n   */\n  supports(type: string): boolean {\n    return (\n      type === 'application/pdf' ||\n      type.endsWith('.pdf') ||\n      type.toLowerCase() === 'pdf'\n    );\n  }\n\n  /**\n   * Process a PDF document\n   *\n   * Extracts text and optionally images/OCR from the PDF, structuring\n   * it into hierarchical document parts.\n   *\n   * @param url - PDF URL or file path\n   * @param options - Processing options\n   * @returns Promise resolving to structured Document\n   */\n  async process(\n    url: string,\n    options: FetchDocumentOptions = {},\n  ): Promise<Document> {\n    // Create and initialize base document\n    const baseDoc = await BaseDocument.create(url, options);\n\n    // Check cache for processed document\n    const cacheKey = `${baseDoc.localPath}.processed_pdf`;\n    const cached = await getCached(cacheKey);\n    if (cached) {\n      try {\n        const parsed = JSON.parse(cached);\n        return {\n          url: baseDoc.url.toString(),\n          type: baseDoc.type,\n          parts: parsed.parts,\n          metadata: parsed.metadata || {},\n        };\n      } catch (error) {\n        // Cache corrupted, continue with fresh processing\n        console.warn('Cached PDF data corrupted, reprocessing', error);\n      }\n    }\n\n    // Validate that the downloaded file is actually a PDF (issue #460, #463)\n    // WordPress Download Manager and some other servers may return HTML\n    // with Content-Type: application/pdf, causing PDF extraction to fail\n    const fileBuffer = await fs.readFile(baseDoc.localPath);\n    const header = fileBuffer.subarray(0, 5).toString('utf-8');\n\n    if (header !== '%PDF-') {\n      // File is not a valid PDF - delete poisoned cache file (issue #463)\n      try {\n        await fs.unlink(baseDoc.localPath);\n      } catch (unlinkError) {\n        console.warn(\n          `Failed to delete poisoned cache file: ${baseDoc.localPath}`,\n          unlinkError,\n        );\n      }\n\n      // Check if it's HTML to provide helpful error message\n      const content = fileBuffer.toString(\n        'utf-8',\n        0,\n        Math.min(1000, fileBuffer.length),\n      );\n      if (content.includes('<!DOCTYPE html>') || content.includes('<html')) {\n        throw new Error(\n          `Downloaded file is HTML, not PDF. The server returned HTML content for ${url}. ` +\n            'This commonly occurs with WordPress Download Manager URLs that return tracking pages. ' +\n            `Expected PDF magic bytes (%PDF-) but got: ${header}. ` +\n            'The poisoned cache file has been removed - please try again.',\n        );\n      } else {\n        throw new Error(\n          `Downloaded file is not a valid PDF. Expected %PDF- magic bytes but got: ${header}. ` +\n            'The invalid cache file has been removed - please try again.',\n        );\n      }\n    }\n\n    // Get PDF reader and extract content\n    const reader = await getPDFReader();\n    const extractedText = await reader.extractText(baseDoc.localPath);\n\n    // Create main document part\n    const mainPart: DocumentPart = {\n      id: uuidv4(),\n      title: getTitleFromUrl(url, 'PDF Document'),\n      content: extractedText || '',\n      type: 'text',\n      metadata: {\n        source: 'pdf',\n        filePath: baseDoc.localPath,\n      },\n    };\n\n    // Extract images if enabled\n    if (options.extractImages === true) {\n      mainPart.images = await this.extractImages(\n        baseDoc.localPath,\n        options.runOcr !== false,\n      );\n    }\n\n    const document: Document = {\n      url: baseDoc.url.toString(),\n      type: baseDoc.type,\n      parts: [mainPart],\n      metadata: {\n        processor: 'pdf',\n        extractedAt: new Date().toISOString(),\n        hasImages: (mainPart.images?.length || 0) > 0,\n      },\n    };\n\n    // Cache the processed document\n    await setCached(cacheKey, JSON.stringify(document));\n\n    return document;\n  }\n\n  /**\n   * Extract images from PDF\n   *\n   * This is a placeholder for future image extraction functionality.\n   * Will use @happyvertical/pdf's image extraction capabilities when available.\n   *\n   * @param filePath - Local PDF file path\n   * @param runOcr - Whether to run OCR on extracted images\n   * @returns Promise resolving to array of DocumentImages\n   */\n  private async extractImages(\n    _filePath: string,\n    _runOcr: boolean,\n  ): Promise<DocumentImage[]> {\n    // TODO: Implement image extraction using @happyvertical/pdf\n    // For now, return empty array as placeholder\n\n    // Future implementation will:\n    // 1. Use getPDFReader() to extract images from PDF\n    // 2. Save images to cache directory\n    // 3. If runOcr is true, use @happyvertical/ocr to extract text from images\n    // 4. Return array of DocumentImage objects with metadata\n\n    return [];\n  }\n}\n\nexport default PDFProcessor;\n","import { getMimeType } from '@happyvertical/files';\nimport { scrapeDocument } from '@happyvertical/spider';\nimport { PDFProcessor } from './processors/pdf';\nimport type { Document, FetchDocumentOptions } from './types';\n\n/**\n * Available document processors\n */\nconst processors = [new PDFProcessor()];\n\n/**\n * Fetch a document from a URL with automatic format detection\n *\n * This factory function:\n * 1. Detects the document format (PDF, HTML, Markdown, etc.)\n * 2. Selects the appropriate processor\n * 3. Processes the document into structured parts\n * 4. Returns a Document object with hierarchical content\n *\n * @param url - Document URL or file path (file://, http://, https://)\n * @param options - Fetch and processing options\n * @returns Promise resolving to structured Document\n *\n * @example\n * ```typescript\n * // Fetch a PDF with image extraction and OCR\n * const doc = await fetchDocument('https://example.com/report.pdf', {\n *   extractImages: true,\n *   runOcr: true\n * });\n *\n * // Access document parts\n * for (const part of doc.parts) {\n *   console.log(part.title);\n *   console.log(part.content);\n *\n *   // Check for images\n *   if (part.images) {\n *     for (const image of part.images) {\n *       console.log(image.url);\n *       console.log(image.ocrText); // Text extracted via OCR\n *     }\n *   }\n * }\n * ```\n */\nexport async function fetchDocument(\n  url: string,\n  options: FetchDocumentOptions = {},\n): Promise<Document> {\n  // For web URLs (http/https), use spider package to detect special cases\n  // (WordPress Download Manager, CivicWeb, DocuShare, etc.)\n  const isWebUrl = url.startsWith('http://') || url.startsWith('https://');\n\n  if (isWebUrl && !options.type) {\n    try {\n      // Use spider to detect WordPress, CivicWeb, DocuShare, and other document management systems\n      const scraped = await scrapeDocument(url, {\n        scraper: options.scraper || 'basic',\n        spider: options.spider || 'dom',\n        cache: options.cache,\n        cacheExpiry: options.cacheExpiry,\n        headers: options.headers,\n        timeout: options.timeout,\n        maxDuration: options.maxDuration,\n        maxInteractions: options.maxInteractions,\n      });\n\n      // Check if spider detected a document management system with PDF link\n      const hasDocLink =\n        scraped.metadata.strategy === 'wordpress-pdf-link' ||\n        scraped.metadata.strategy === 'civicweb-pdf-link' ||\n        scraped.metadata.strategy === 'docushare-pdf-link';\n\n      if (hasDocLink && scraped.metadata.isPdf && !scraped.metadata.complete) {\n        // Spider detected a document management page and extracted the PDF URL\n        // Use the extracted URL for PDF processing\n        url = scraped.url;\n        options.type = 'application/pdf';\n      }\n    } catch (error) {\n      // If spider fails, continue with direct download\n      // This ensures backward compatibility\n      console.warn(\n        `Spider detection failed for ${url}, falling back to direct download:`,\n        error,\n      );\n    }\n  }\n\n  // Determine type - check URL extension first, then MIME type\n  // This handles servers that return incorrect Content-Type headers (e.g., application/octet-stream for PDFs)\n  let type = options.type;\n\n  if (!type) {\n    // Extract file extension from URL\n    const urlLower = url.toLowerCase();\n\n    // Check for common document extensions in URL\n    if (\n      urlLower.endsWith('.pdf') ||\n      urlLower.includes('.pdf?') ||\n      urlLower.includes('.pdf#')\n    ) {\n      type = 'application/pdf';\n    } else {\n      // Fall back to MIME type detection\n      type = getMimeType(url) || '';\n    }\n  }\n\n  // Find appropriate processor\n  const processor = processors.find((p) => p.supports(type));\n\n  if (!processor) {\n    throw new Error(\n      `No processor available for document type: ${type}. Supported types: PDF (.pdf, application/pdf)`,\n    );\n  }\n\n  // Process document\n  return processor.process(url, options);\n}\n\nexport default fetchDocument;\n","/**\n * @happyvertical/documents - Document processing with multi-part structure\n *\n * Provides document processing for PDFs with support for:\n * - Hierarchical document parts\n * - Automatic format detection from URL or MIME type\n * - Document management system detection (WordPress, CivicWeb, DocuShare)\n * - File caching for performance\n *\n * @example\n * ```typescript\n * import { fetchDocument } from '@happyvertical/documents';\n *\n * const doc = await fetchDocument('https://example.com/report.pdf');\n *\n * for (const part of doc.parts) {\n *   console.log(part.title);\n *   console.log(part.content);\n * }\n * ```\n */\n\n// Base classes\nexport { Document } from './document';\n// Main factory function\nexport { fetchDocument } from './factory';\n\n// Processors\nexport { PDFProcessor } from './processors/pdf';\n// Types\nexport type {\n  Document as DocumentType,\n  DocumentImage,\n  DocumentPart,\n  DocumentProcessor,\n  FetchDocumentOptions,\n} from './types';\n// Utilities\nexport { getTitleFromUrl } from './utils';\n\n/** @internal */\nexport const PACKAGE_VERSION_INITIALIZED = true;\n"],"names":["URL","BaseDocument","fs","uuidv4"],"mappings":";;;;;;;;;AAkBO,MAAM,SAAS;AAAA;AAAA;AAAA;AAAA,EAIV,WAAW;AAAA;AAAA;AAAA;AAAA,EAKX;AAAA;AAAA;AAAA;AAAA,EAKF,aAAa;AAAA;AAAA;AAAA;AAAA,EAKb,YAAY;AAAA;AAAA;AAAA;AAAA,EAKb;AAAA;AAAA;AAAA;AAAA,EAKA;AAAA;AAAA;AAAA;AAAA,EAKA,QAAwB,CAAA;AAAA;AAAA;AAAA;AAAA,EAKxB,WAAgC,CAAA;AAAA;AAAA;AAAA;AAAA,EAKvC,IAAW,YAAoB;AAC7B,WAAO,KAAK;AAAA,EACd;AAAA;AAAA;AAAA;AAAA,EAKA,IAAW,WAAmB;AAC5B,WAAO,KAAK;AAAA,EACd;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAQA,YAAY,KAAa,UAAgC,IAAI;AAC3D,SAAK,MAAM,IAAIA,MAAI,GAAG;AACtB,SAAK,UAAU;AACf,SAAK,OACH,QAAQ,QAAQ,YAAY,KAAK,IAAI,SAAA,CAAU,KAAK;AAEtD,SAAK,YACH,QAAQ,YACR,KAAK,QAAQ,GAAG,OAAA,GAAU,UAAU,YAAY,WAAW;AAE7D,QAAI,KAAK,IAAI,SAAS,WAAW,MAAM,GAAG;AAGxC,WAAK,aAAa,mBAAmB,KAAK,IAAI,QAAQ;AACtD,WAAK,WAAW;AAAA,IAClB,WAAW,KAAK,IAAI,SAAS,WAAW,MAAM,GAAG;AAG/C,UAAI,WAAW,KAAK,IAAI;AAGxB,UAAI,SAAS,SAAS,GAAG,GAAG;AAC1B,mBAAW,SAAS,MAAM,GAAG,EAAE;AAAA,MACjC;AAIA,UAAI,CAAC,SAAS,MAAM,eAAe,GAAG;AAEpC,YACE,KAAK,SAAS,qBACd,QAAQ,SAAS,mBACjB;AACA,sBAAY;AAAA,QACd;AAAA,MAEF;AAEA,WAAK,aAAa,KAAK;AAAA,QACrB,KAAK;AAAA,QACL,SAAS,KAAK,IAAI,QAAQ;AAAA,QAC1B;AAAA,MAAA;AAEF,WAAK,WAAW;AAAA,IAClB;AAAA,EACF;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAWA,aAAa,OACX,KACA,UAAgC,IACb;AACnB,UAAM,WAAW,IAAI,SAAS,KAAK,OAAO;AAC1C,UAAM,SAAS,WAAA;AACf,WAAO;AAAA,EACT;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAOA,MAAM,aAA4B;AAChC,QAAI,KAAK,UAAU;AACjB,UAAI,CAAC,KAAK,KAAK;AACb,cAAM,IAAI,MAAM,oDAAoD;AAAA,MACtE;AACA,YAAM,sBAAsB,KAAK,IAAI,SAAA,GAAY,KAAK,UAAU;AAAA,IAClE;AAAA,EACF;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAOO,aAAsB;AAC3B,QAAI,CAAC,KAAK,KAAM,QAAO;AAEvB,WACE,KAAK,KAAK,WAAW,OAAO,KAC5B,KAAK,SAAS,sBACd,KAAK,SAAS,qBACd,KAAK,SAAS,4BACd,KAAK,SAAS,4BACd;AAAA,MACE;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,MACA;AAAA,IAAA,EACA,KAAK,CAAC,QAAQ,KAAK,UAAU,YAAA,EAAc,SAAS,GAAG,CAAC;AAAA,EAE9D;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAOO,aAA2B;AAChC,WAAO;AAAA,MACL,KAAK,KAAK,IAAI,SAAA;AAAA,MACd,MAAM,KAAK;AAAA,MACX,OAAO,KAAK;AAAA,MACZ,UAAU,KAAK;AAAA,IAAA;AAAA,EAEnB;AACF;AChLO,SAAS,gBACd,KACA,eAAe,YACP;AACR,MAAI;AACF,UAAM,SAAS,IAAI,IAAI,GAAG;AAC1B,UAAM,WAAW,OAAO;AACxB,UAAM,WAAW,SAAS,MAAM,GAAG,EAAE,SAAS;AAG9C,UAAM,kBAAkB,mBAAmB,QAAQ;AAGnD,WAAO,gBACJ,QAAQ,0BAA0B,EAAE,EACpC,QAAQ,SAAS,GAAG,EACpB,KAAA;AAAA,EACL,QAAQ;AACN,WAAO;AAAA,EACT;AACF;ACpBO,MAAM,aAA0C;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAQrD,SAAS,MAAuB;AAC9B,WACE,SAAS,qBACT,KAAK,SAAS,MAAM,KACpB,KAAK,kBAAkB;AAAA,EAE3B;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAYA,MAAM,QACJ,KACA,UAAgC,IACb;AAEnB,UAAM,UAAU,MAAMC,SAAa,OAAO,KAAK,OAAO;AAGtD,UAAM,WAAW,GAAG,QAAQ,SAAS;AACrC,UAAM,SAAS,MAAM,UAAU,QAAQ;AACvC,QAAI,QAAQ;AACV,UAAI;AACF,cAAM,SAAS,KAAK,MAAM,MAAM;AAChC,eAAO;AAAA,UACL,KAAK,QAAQ,IAAI,SAAA;AAAA,UACjB,MAAM,QAAQ;AAAA,UACd,OAAO,OAAO;AAAA,UACd,UAAU,OAAO,YAAY,CAAA;AAAA,QAAC;AAAA,MAElC,SAAS,OAAO;AAEd,gBAAQ,KAAK,2CAA2C,KAAK;AAAA,MAC/D;AAAA,IACF;AAKA,UAAM,aAAa,MAAMC,SAAG,SAAS,QAAQ,SAAS;AACtD,UAAM,SAAS,WAAW,SAAS,GAAG,CAAC,EAAE,SAAS,OAAO;AAEzD,QAAI,WAAW,SAAS;AAEtB,UAAI;AACF,cAAMA,SAAG,OAAO,QAAQ,SAAS;AAAA,MACnC,SAAS,aAAa;AACpB,gBAAQ;AAAA,UACN,yCAAyC,QAAQ,SAAS;AAAA,UAC1D;AAAA,QAAA;AAAA,MAEJ;AAGA,YAAM,UAAU,WAAW;AAAA,QACzB;AAAA,QACA;AAAA,QACA,KAAK,IAAI,KAAM,WAAW,MAAM;AAAA,MAAA;AAElC,UAAI,QAAQ,SAAS,iBAAiB,KAAK,QAAQ,SAAS,OAAO,GAAG;AACpE,cAAM,IAAI;AAAA,UACR,0EAA0E,GAAG,qIAE9B,MAAM;AAAA,QAAA;AAAA,MAGzD,OAAO;AACL,cAAM,IAAI;AAAA,UACR,2EAA2E,MAAM;AAAA,QAAA;AAAA,MAGrF;AAAA,IACF;AAGA,UAAM,SAAS,MAAM,aAAA;AACrB,UAAM,gBAAgB,MAAM,OAAO,YAAY,QAAQ,SAAS;AAGhE,UAAM,WAAyB;AAAA,MAC7B,IAAIC,GAAA;AAAA,MACJ,OAAO,gBAAgB,KAAK,cAAc;AAAA,MAC1C,SAAS,iBAAiB;AAAA,MAC1B,MAAM;AAAA,MACN,UAAU;AAAA,QACR,QAAQ;AAAA,QACR,UAAU,QAAQ;AAAA,MAAA;AAAA,IACpB;AAIF,QAAI,QAAQ,kBAAkB,MAAM;AAClC,eAAS,SAAS,MAAM,KAAK;AAAA,QAC3B,QAAQ;AAAA,QACR,QAAQ,WAAW;AAAA,MAAA;AAAA,IAEvB;AAEA,UAAM,WAAqB;AAAA,MACzB,KAAK,QAAQ,IAAI,SAAA;AAAA,MACjB,MAAM,QAAQ;AAAA,MACd,OAAO,CAAC,QAAQ;AAAA,MAChB,UAAU;AAAA,QACR,WAAW;AAAA,QACX,cAAa,oBAAI,KAAA,GAAO,YAAA;AAAA,QACxB,YAAY,SAAS,QAAQ,UAAU,KAAK;AAAA,MAAA;AAAA,IAC9C;AAIF,UAAM,UAAU,UAAU,KAAK,UAAU,QAAQ,CAAC;AAElD,WAAO;AAAA,EACT;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAYA,MAAc,cACZ,WACA,SAC0B;AAU1B,WAAO,CAAA;AAAA,EACT;AACF;AC3KA,MAAM,aAAa,CAAC,IAAI,cAAc;AAsCtC,eAAsB,cACpB,KACA,UAAgC,IACb;AAGnB,QAAM,WAAW,IAAI,WAAW,SAAS,KAAK,IAAI,WAAW,UAAU;AAEvE,MAAI,YAAY,CAAC,QAAQ,MAAM;AAC7B,QAAI;AAEF,YAAM,UAAU,MAAM,eAAe,KAAK;AAAA,QACxC,SAAS,QAAQ,WAAW;AAAA,QAC5B,QAAQ,QAAQ,UAAU;AAAA,QAC1B,OAAO,QAAQ;AAAA,QACf,aAAa,QAAQ;AAAA,QACrB,SAAS,QAAQ;AAAA,QACjB,SAAS,QAAQ;AAAA,QACjB,aAAa,QAAQ;AAAA,QACrB,iBAAiB,QAAQ;AAAA,MAAA,CAC1B;AAGD,YAAM,aACJ,QAAQ,SAAS,aAAa,wBAC9B,QAAQ,SAAS,aAAa,uBAC9B,QAAQ,SAAS,aAAa;AAEhC,UAAI,cAAc,QAAQ,SAAS,SAAS,CAAC,QAAQ,SAAS,UAAU;AAGtE,cAAM,QAAQ;AACd,gBAAQ,OAAO;AAAA,MACjB;AAAA,IACF,SAAS,OAAO;AAGd,cAAQ;AAAA,QACN,+BAA+B,GAAG;AAAA,QAClC;AAAA,MAAA;AAAA,IAEJ;AAAA,EACF;AAIA,MAAI,OAAO,QAAQ;AAEnB,MAAI,CAAC,MAAM;AAET,UAAM,WAAW,IAAI,YAAA;AAGrB,QACE,SAAS,SAAS,MAAM,KACxB,SAAS,SAAS,OAAO,KACzB,SAAS,SAAS,OAAO,GACzB;AACA,aAAO;AAAA,IACT,OAAO;AAEL,aAAO,YAAY,GAAG,KAAK;AAAA,IAC7B;AAAA,EACF;AAGA,QAAM,YAAY,WAAW,KAAK,CAAC,MAAM,EAAE,SAAS,IAAI,CAAC;AAEzD,MAAI,CAAC,WAAW;AACd,UAAM,IAAI;AAAA,MACR,6CAA6C,IAAI;AAAA,IAAA;AAAA,EAErD;AAGA,SAAO,UAAU,QAAQ,KAAK,OAAO;AACvC;ACjFO,MAAM,8BAA8B;"}

package/dist/processors/pdf.d.ts

@@ -0,0 +1,45 @@
+import { Document, DocumentProcessor, FetchDocumentOptions } from '../types';
+/**
+ * PDF Document Processor
+ *
+ * Handles PDF documents with support for:
+ * - Text extraction from PDF content via `@happyvertical/pdf`
+ * - PDF header validation (detects HTML cache poisoning from document management systems)
+ * - Processed document caching via `@happyvertical/files`
+ *
+ * Image extraction and OCR are stubbed for future implementation.
+ */
+export declare class PDFProcessor implements DocumentProcessor {
+    /**
+     * Check if this processor supports the given MIME type or extension.
+     * Accepts `'application/pdf'`, `'.pdf'`, or `'pdf'` (case-insensitive).
+     *
+     * @param type - MIME type or file extension to check
+     * @returns `true` if this processor can handle the given type
+     */
+    supports(type: string): boolean;
+    /**
+     * Process a PDF document
+     *
+     * Extracts text and optionally images/OCR from the PDF, structuring
+     * it into hierarchical document parts.
+     *
+     * @param url - PDF URL or file path
+     * @param options - Processing options
+     * @returns Promise resolving to structured Document
+     */
+    process(url: string, options?: FetchDocumentOptions): Promise<Document>;
+    /**
+     * Extract images from PDF
+     *
+     * This is a placeholder for future image extraction functionality.
+     * Will use @happyvertical/pdf's image extraction capabilities when available.
+     *
+     * @param filePath - Local PDF file path
+     * @param runOcr - Whether to run OCR on extracted images
+     * @returns Promise resolving to array of DocumentImages
+     */
+    private extractImages;
+}
+export default PDFProcessor;
+//# sourceMappingURL=pdf.d.ts.map

package/dist/processors/pdf.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"pdf.d.ts","sourceRoot":"","sources":["../../src/processors/pdf.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EACV,QAAQ,EAGR,iBAAiB,EACjB,oBAAoB,EACrB,MAAM,UAAU,CAAC;AAGlB;;;;;;;;;GASG;AACH,qBAAa,YAAa,YAAW,iBAAiB;IACpD;;;;;;OAMG;IACH,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO;IAQ/B;;;;;;;;;OASG;IACG,OAAO,CACX,GAAG,EAAE,MAAM,EACX,OAAO,GAAE,oBAAyB,GACjC,OAAO,CAAC,QAAQ,CAAC;IAqGpB;;;;;;;;;OASG;YACW,aAAa;CAe5B;AAED,eAAe,YAAY,CAAC"}

package/dist/types.d.ts

@@ -0,0 +1,199 @@
+/**
+ * Type definitions for the @happyvertical/documents package
+ */
+/**
+ * Image extracted from a document
+ *
+ * Represents an image found in a document (PDF, HTML, etc.)
+ * with optional OCR text extraction for scanned images.
+ */
+export interface DocumentImage {
+    /**
+     * Unique identifier for the image
+     */
+    id: string;
+    /**
+     * URL or reference to the image
+     */
+    url: string;
+    /**
+     * Local filesystem path if image has been downloaded
+     */
+    localPath?: string;
+    /**
+     * Alt text from HTML or PDF metadata
+     */
+    altText?: string;
+    /**
+     * Text extracted from image via OCR
+     * Useful for scanned documents or images with text
+     */
+    ocrText?: string;
+    /**
+     * Position/order of image in the document
+     */
+    position?: number;
+    /**
+     * Image metadata (dimensions, format, etc.)
+     */
+    metadata?: {
+        width?: number;
+        height?: number;
+        format?: string;
+        [key: string]: any;
+    };
+}
+/**
+ * A part or section of a document
+ *
+ * Documents can be hierarchical with nested parts (e.g., sections, chapters).
+ * Each part contains content, optional images, and can have child parts.
+ */
+export interface DocumentPart {
+    /**
+     * Unique identifier for this part
+     */
+    id: string;
+    /**
+     * Title or heading for this part
+     */
+    title: string;
+    /**
+     * Text content of this part
+     */
+    content: string;
+    /**
+     * Content type for this part
+     */
+    type: 'text' | 'html' | 'markdown';
+    /**
+     * Images contained in this part
+     */
+    images?: DocumentImage[];
+    /**
+     * Additional metadata for this part
+     */
+    metadata?: Record<string, any>;
+    /**
+     * Nested child parts (for hierarchical documents)
+     */
+    parts?: DocumentPart[];
+}
+/**
+ * Complete document with all parts and metadata
+ */
+export interface Document {
+    /**
+     * Source URL of the document
+     */
+    url: string;
+    /**
+     * MIME type or document type
+     */
+    type: string;
+    /**
+     * Document parts (can be hierarchical)
+     */
+    parts: DocumentPart[];
+    /**
+     * Document-level metadata
+     */
+    metadata?: Record<string, any>;
+}
+/**
+ * Options for fetching a document
+ */
+export interface FetchDocumentOptions {
+    /**
+     * Directory for caching downloaded files
+     * @default os.tmpdir()/.cache/have-sdk/documents
+     */
+    cacheDir?: string;
+    /**
+     * Whether to extract images from the document
+     * @default true
+     */
+    extractImages?: boolean;
+    /**
+     * Whether to run OCR on images (PDF scans, etc.)
+     * @default true for PDFs, false for HTML/Markdown
+     */
+    runOcr?: boolean;
+    /**
+     * Scraper type to use for content extraction
+     * - 'basic': Fast, static HTML scraping (default)
+     * - 'crawlee': Full browser with JavaScript execution
+     * @default 'basic'
+     */
+    scraper?: 'basic' | 'crawlee';
+    /**
+     * Spider adapter to use for fetching web pages
+     * - 'simple': Basic HTTP fetch
+     * - 'dom': HTML parsing with happy-dom
+     * - 'crawlee': Headless browser (requires scraper: 'crawlee')
+     * @default 'dom'
+     */
+    spider?: 'simple' | 'dom' | 'crawlee';
+    /**
+     * Spider adapter to use for HTML fetching (deprecated, use 'spider' instead)
+     * @default 'simple'
+     * @deprecated Use 'spider' instead
+     */
+    spiderAdapter?: 'simple' | 'dom' | 'crawlee';
+    /**
+     * Whether to use cache for spider fetching
+     * @default true
+     */
+    cache?: boolean;
+    /**
+     * Cache expiry time in milliseconds for spider fetching
+     * @default 300000 (5 minutes)
+     */
+    cacheExpiry?: number;
+    /**
+     * Custom HTTP headers for spider requests
+     */
+    headers?: Record<string, string>;
+    /**
+     * Request timeout in milliseconds for spider fetching
+     * @default 30000 (30 seconds)
+     */
+    timeout?: number;
+    /**
+     * Maximum time to spend scraping in milliseconds
+     * Used by advanced scrapers (tree, pagination, etc.)
+     */
+    maxDuration?: number;
+    /**
+     * Maximum number of interactions to perform
+     * Used by advanced scrapers (clicking, scrolling, etc.)
+     */
+    maxInteractions?: number;
+    /**
+     * Override MIME type detection
+     */
+    type?: string;
+}
+/**
+ * Interface for document processors
+ *
+ * Each processor handles a specific document format (PDF, HTML, Markdown, etc.)
+ */
+export interface DocumentProcessor {
+    /**
+     * Process a document and return structured parts
+     *
+     * @param url - Source URL or file path
+     * @param options - Processing options
+     * @returns Promise resolving to Document with parts
+     */
+    process(url: string, options?: FetchDocumentOptions): Promise<Document>;
+    /**
+     * Check if this processor can handle the given type
+     *
+     * @param type - MIME type or file extension
+     * @returns True if processor supports this type
+     */
+    supports(type: string): boolean;
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B;;OAEG;IACH,EAAE,EAAE,MAAM,CAAC;IAEX;;OAEG;IACH,GAAG,EAAE,MAAM,CAAC;IAEZ;;OAEG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IAEnB;;OAEG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;IAEjB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;IAEjB;;OAEG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAElB;;OAEG;IACH,QAAQ,CAAC,EAAE;QACT,KAAK,CAAC,EAAE,MAAM,CAAC;QACf,MAAM,CAAC,EAAE,MAAM,CAAC;QAChB,MAAM,CAAC,EAAE,MAAM,CAAC;QAChB,CAAC,GAAG,EAAE,MAAM,GAAG,GAAG,CAAC;KACpB,CAAC;CACH;AAED;;;;;GAKG;AACH,MAAM,WAAW,YAAY;IAC3B;;OAEG;IACH,EAAE,EAAE,MAAM,CAAC;IAEX;;OAEG;IACH,KAAK,EAAE,MAAM,CAAC;IAEd;;OAEG;IACH,OAAO,EAAE,MAAM,CAAC;IAEhB;;OAEG;IACH,IAAI,EAAE,MAAM,GAAG,MAAM,GAAG,UAAU,CAAC;IAEnC;;OAEG;IACH,MAAM,CAAC,EAAE,aAAa,EAAE,CAAC;IAEzB;;OAEG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAE/B;;OAEG;IACH,KAAK,CAAC,EAAE,YAAY,EAAE,CAAC;CACxB;AAED;;GAEG;AACH,MAAM,WAAW,QAAQ;IACvB;;OAEG;IACH,GAAG,EAAE,MAAM,CAAC;IAEZ;;OAEG;IACH,IAAI,EAAE,MAAM,CAAC;IAEb;;OAEG;IACH,KAAK,EAAE,YAAY,EAAE,CAAC;IAEtB;;OAEG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;CAChC;AAED;;GAEG;AACH,MAAM,WAAW,oBAAoB;IACnC;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAElB;;;OAGG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;IAExB;;;OAGG;IACH,MAAM,CAAC,EAAE,OAAO,CAAC;IAEjB;;;;;OAKG;IACH,OAAO,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IAE9B;;;;;;OAMG;IACH,MAAM,CAAC,EAAE,QAAQ,GAAG,KAAK,GAAG,SAAS,CAAC;IAEtC;;;;OAIG;IACH,aAAa,CAAC,EAAE,QAAQ,GAAG,KAAK,GAAG,SAAS,CAAC;IAE7C;;;OAGG;IACH,KAAK,CAAC,EAAE,OAAO,CAAC;IAEhB;;;OAGG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IAErB;;OAEG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAEjC;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;IAEjB;;;OAGG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IAErB;;;OAGG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IAEzB;;OAEG;IACH,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED;;;;GAIG;AACH,MAAM,WAAW,iBAAiB;IAChC;;;;;;OAMG;IACH,OAAO,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC;IAExE;;;;;OAKG;IACH,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC;CACjC"}

package/dist/utils.d.ts

@@ -0,0 +1,25 @@
+/**
+ * Utility functions for document processing
+ */
+/**
+ * Extract a human-readable title from a URL
+ *
+ * Takes a URL and extracts the filename from the pathname, then formats it
+ * into a readable title by removing the extension and converting separators
+ * to spaces. Also decodes URL-encoded characters like %20.
+ *
+ * @param url - URL string to extract title from
+ * @param defaultTitle - Default title to use if extraction fails
+ * @returns Formatted title string
+ *
+ * @example
+ * ```typescript
+ * getTitleFromUrl('file:///path/to/My%20Document.pdf')
+ * // Returns: 'My Document'
+ *
+ * getTitleFromUrl('https://example.com/research_paper.pdf')
+ * // Returns: 'research paper'
+ * ```
+ */
+export declare function getTitleFromUrl(url: string, defaultTitle?: string): string;
+//# sourceMappingURL=utils.d.ts.map

package/dist/utils.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"utils.d.ts","sourceRoot":"","sources":["../src/utils.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH;;;;;;;;;;;;;;;;;;;GAmBG;AACH,wBAAgB,eAAe,CAC7B,GAAG,EAAE,MAAM,EACX,YAAY,SAAa,GACxB,MAAM,CAiBR"}

package/metadata.json

@@ -0,0 +1,35 @@
+{
+  "name": "@happyvertical/documents",
+  "path": "packages/documents",
+  "position": {
+    "index": 8,
+    "count": 30
+  },
+  "description": "Multi-part document processing with support for PDF, HTML, and Markdown",
+  "provides": [
+    "Multi-part document processing with support for PDF, HTML, and Markdown"
+  ],
+  "implements": [],
+  "requires": {
+    "workspace": [
+      "@happyvertical/files",
+      "@happyvertical/utils"
+    ],
+    "externalHappyVertical": [
+      "@happyvertical/ocr",
+      "@happyvertical/pdf",
+      "@happyvertical/spider"
+    ],
+    "external": [
+      "uuid"
+    ]
+  },
+  "dependents": [],
+  "stability": {
+    "level": "stable",
+    "reason": "Primary package surface is described as implemented and production-oriented."
+  },
+  "keywords": [
+    "documents"
+  ]
+}

package/package.json

@@ -0,0 +1,67 @@
+{
+  "name": "@happyvertical/documents",
+  "version": "0.74.9",
+  "description": "Multi-part document processing with support for PDF, HTML, and Markdown",
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "bin": {
+    "have-documents-context": "./dist/cli/claude-context.js"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE",
+    "AGENT.md",
+    "metadata.json"
+  ],
+  "publishConfig": {
+    "registry": "https://registry.npmjs.org",
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/happyvertical/sdk.git",
+    "directory": "packages/documents"
+  },
+  "bugs": {
+    "url": "https://github.com/happyvertical/sdk/issues"
+  },
+  "homepage": "https://github.com/happyvertical/sdk/tree/main/packages/documents#readme",
+  "license": "MIT",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "dependencies": {
+    "@happyvertical/ocr": "^0.60.39",
+    "@happyvertical/pdf": "^0.62.25",
+    "@happyvertical/spider": "^0.60.10",
+    "uuid": "^13.0.0",
+    "@happyvertical/files": "0.74.9",
+    "@happyvertical/utils": "0.74.9"
+  },
+  "devDependencies": {
+    "@types/node": "25.0.10",
+    "typescript": "^5.9.3",
+    "vite": "7.3.2",
+    "vitest": "^4.1.5"
+  },
+  "keywords": [
+    "documents",
+    "pdf",
+    "html",
+    "markdown",
+    "ocr",
+    "content-extraction"
+  ],
+  "scripts": {
+    "build": "vite build",
+    "build:watch": "vite build --watch",
+    "dev": "vite --port 3004",
+    "test": "vitest run",
+    "test:watch": "vitest watch"
+  }
+}