@dragon708/docmind-markdown 1.2.1 → 1.2.3

package/node_modules/unpdf/README.md

@@ -0,0 +1,371 @@
+# unpdf
+
+A collection of utilities for PDF extraction and rendering. Designed specifically for serverless environments, but it also works in Node.js, Deno, Bun and the browser. `unpdf` is particularly useful for serverless AI applications, especially for summarizing PDF documents in document analysis workflows.
+
+This library ships with a serverless build/redistribution of Mozilla's [PDF.js](https://github.com/mozilla/pdf.js) that is optimized for edge environments. Some string replacements, global mocks and inlining the PDF.js worker allow the browser code to become platform agnostic. See [`pdfjs.rollup.config.ts`](./pdfjs.rollup.config.ts) for the details.
+
+This library is also intended as a modern alternative to the unmaintained but still popular [`pdf-parse`](https://www.npmjs.com/package/pdf-parse).
+
+## Features
+
+- 🏗️ Made for Node.js, browser and serverless environments
+- 🪭 Includes serverless build of PDF.js ([`unpdf/pdfjs`](./package.json#L34))
+- 💬 Extract [text](#extract-text-from-pdf), [links](#extractlinks), and [images](#extractimages) from PDF files
+- 🧠 Perfect for AI applications and PDF summarization
+- 🧱 Opt-in to legacy PDF.js build
+- 💨 Zero dependencies
+
+## PDF.js Compatibility
+
+> [!Tip]
+> The serverless PDF.js bundle provided by `unpdf` is built from PDF.js v5.4.296.
+
+You can use an [official PDF.js build](#official-or-legacy-pdfjs-build) by using the [`definePDFJSModule`](#definepdfjsmodule) method. This is useful if you want to use a specific version or a custom build of PDF.js.
+
+## Installation
+
+Run the following command to add `unpdf` to your project.
+
+```bash
+# pnpm
+pnpm add -D unpdf
+
+# npm
+npm install -D unpdf
+
+# yarn
+yarn add -D unpdf
+```
+
+## Usage
+
+### Extract Text From PDF
+
+```ts
+import { extractText, getDocumentProxy } from 'unpdf'
+
+// Either fetch a PDF file from the web or load it from the file system
+const buffer = await fetch('https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf')
+  .then(res => res.arrayBuffer())
+const buffer = await readFile('./dummy.pdf')
+
+// Then, load the PDF file into a PDF.js document
+const pdf = await getDocumentProxy(new Uint8Array(buffer))
+
+// Finally, extract the text from the PDF file
+const { totalPages, text } = await extractText(pdf, { mergePages: true })
+
+console.log(`Total pages: ${totalPages}`)
+console.log(text)
+```
+
+### Official or Legacy PDF.js Build
+
+Usually you don't need to worry about the PDF.js build. `unpdf` ships with a serverless build of the latest PDF.js version. However, if you want to use the official PDF.js version or the legacy build, you can define a custom PDF.js module.
+
+> [!WARNING]
+> PDF.js v5.x uses `Promise.withResolvers`, which may not be supported in all environments, such as Node < 22. Consider to use the bundled serverless build, which includes a polyfill, or use an older version of PDF.js.
+
+For example, if you want to use the official PDF.js build, you can do the following:
+
+```ts
+import { definePDFJSModule, extractText, getDocumentProxy } from 'unpdf'
+
+// Define the PDF.js build before using any other unpdf method
+await definePDFJSModule(() => import('pdfjs-dist'))
+
+// Now, you can use all unpdf methods with the official PDF.js build
+const pdf = await getDocumentProxy(/* … */)
+const { text } = await extractText(pdf)
+```
+
+### PDF.js API
+
+`unpdf` provides helpful [methods](#api) to work with PDF files, such as `extractText` and `extractImages`, which should cover most use cases. However, if you need more control over the PDF.js API, you can use the `getResolvedPDFJS` method to get the resolved PDF.js module.
+
+Access the PDF.js API directly by calling `getResolvedPDFJS`:
+
+```ts
+import { getResolvedPDFJS } from 'unpdf'
+
+const { version } = await getResolvedPDFJS()
+```
+
+> [!NOTE]
+> If no other PDF.js build was defined, the serverless build will always be used.
+
+For example, you can use the `getDocument` method to load a PDF file and then use the `getMetadata` method to get the metadata of the PDF file:
+
+```ts
+import { readFile } from 'node:fs/promises'
+import { getResolvedPDFJS } from 'unpdf'
+
+const { getDocument } = await getResolvedPDFJS()
+const data = await readFile('./dummy.pdf')
+const document = await getDocument(new Uint8Array(data)).promise
+
+console.log(await document.getMetadata())
+```
+
+## API
+
+### `definePDFJSModule`
+
+Allows to define a custom PDF.js build. This method should be called before using any other method. If no custom build is defined, the serverless build will be used.
+
+**Type Declaration**
+
+```ts
+function definePDFJSModule(pdfjs: () => Promise<PDFJS>): Promise<void>
+```
+
+### `getResolvedPDFJS`
+
+Returns the resolved PDF.js module. If no other PDF.js build was defined, the serverless build will be used. This method is useful if you want to use the PDF.js API directly.
+
+**Type Declaration**
+
+```ts
+function getResolvedPDFJS(): Promise<PDFJS>
+```
+
+### `getMeta`
+
+Extracts metadata from a PDF. If `parseDates` is set to `true`, the date properties will be parsed into `Date` objects.
+
+**Type Declaration**
+
+```ts
+function getMeta(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  options?: {
+    parseDates?: boolean
+  },
+): Promise<{
+  info: Record<string, any>
+  metadata: Record<string, any>
+}>
+```
+
+### `extractText`
+
+Extracts all text from a PDF. If `mergePages` is set to `true`, the text of all pages will be merged into a single string. Otherwise, an array of strings for each page will be returned.
+
+**Type Declaration**
+
+```ts
+function extractText(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  options?: {
+    mergePages?: false
+  }
+): Promise<{
+  totalPages: number
+  text: string[]
+}>
+function extractText(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  options: {
+    mergePages: true
+  }
+): Promise<{
+  totalPages: number
+  text: string
+}>
+```
+
+### `extractLinks`
+
+Extracts all links from a PDF document, including hyperlinks and external URLs.
+
+**Type Declaration**
+
+```ts
+function extractLinks(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+): Promise<{
+  totalPages: number
+  links: string[]
+}>
+```
+
+**Example**
+
+```ts
+import { readFile } from 'node:fs/promises'
+import { extractLinks, getDocumentProxy } from 'unpdf'
+
+// Load a PDF file
+const buffer = await readFile('./document.pdf')
+const pdf = await getDocumentProxy(new Uint8Array(buffer))
+
+// Extract all links from the PDF
+const { totalPages, links } = await extractLinks(pdf)
+
+console.log(`Total pages: ${totalPages}`)
+console.log(`Found ${links.length} links:`)
+for (const link of links) console.log(link)
+```
+
+### `extractImages`
+
+Extracts images from a specific page of a PDF document, including necessary metadata such as width, height, and calculated color channels.
+
+> [!NOTE]
+> This method will only work in Node.js and browser environments.
+
+In order to use this method, make sure to meet the following requirements:
+
+- Use the official PDF.js build (see below for details).
+- Install the [`@napi-rs/canvas`](https://github.com/Brooooooklyn/canvas) package if you are using Node.js. This package is required to render the PDF page as an image.
+
+**Type Declaration**
+
+```ts
+interface ExtractedImageObject {
+  data: Uint8ClampedArray
+  width: number
+  height: number
+  channels: 1 | 3 | 4
+  key: string
+}
+
+function extractImages(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  pageNumber: number,
+): Promise<ExtractedImageObject[]>
+```
+
+**Example**
+
+> [!NOTE]
+> The following example uses the [sharp](https://github.com/lovell/sharp) library to process and save the extracted images. You will need to install it with your preferred package manager.
+
+```ts
+import { readFile, writeFile } from 'node:fs/promises'
+import sharp from 'sharp'
+import { extractImages, getDocumentProxy } from 'unpdf'
+
+async function extractPdfImages() {
+  const buffer = await readFile('./document.pdf')
+  const pdf = await getDocumentProxy(new Uint8Array(buffer))
+
+  // Extract images from page 1
+  const imagesData = await extractImages(pdf, 1)
+  console.log(`Found ${imagesData.length} images on page 1`)
+
+  // Process each image with sharp (optional)
+  let totalImagesProcessed = 0
+  for (const imgData of imagesData) {
+    const imageIndex = ++totalImagesProcessed
+
+    await sharp(imgData.data, {
+      raw: {
+        width: imgData.width,
+        height: imgData.height,
+        channels: imgData.channels
+      }
+    })
+      .png()
+      .toFile(`image-${imageIndex}.png`)
+
+    console.log(`Saved image ${imageIndex} (${imgData.width}x${imgData.height}, ${imgData.channels} channels)`)
+  }
+}
+
+extractPdfImages().catch(console.error)
+```
+
+### `renderPageAsImage`
+
+To render a PDF page as an image, you can use the `renderPageAsImage` method. This method will return an `ArrayBuffer` of the rendered image. It can also return a data URL (`string`) if `toDataURL` option is set to `true`.
+
+> [!NOTE]
+> This method will only work in Node.js and browser environments.
+
+In order to use this method, make sure to meet the following requirements:
+
+- Use the official PDF.js build (see below for details).
+- Install the [`@napi-rs/canvas`](https://github.com/Brooooooklyn/canvas) package if you are using Node.js. This package is required to render the PDF page as an image.
+
+**Type Declaration**
+
+```ts
+function renderPageAsImage(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  pageNumber: number,
+  options?: {
+    canvasImport?: () => Promise<typeof import('@napi-rs/canvas')>
+    /** @default 1.0 */
+    scale?: number
+    width?: number
+    height?: number
+    toDataURL?: false
+  },
+): Promise<ArrayBuffer>
+function renderPageAsImage(
+  data: DocumentInitParameters['data'] | PDFDocumentProxy,
+  pageNumber: number,
+  options: {
+    canvasImport?: () => Promise<typeof import('@napi-rs/canvas')>
+    /** @default 1.0 */
+    scale?: number
+    width?: number
+    height?: number
+    toDataURL: true
+  },
+): Promise<string>
+```
+
+**Examples**
+
+```ts
+import { definePDFJSModule, renderPageAsImage } from 'unpdf'
+
+// Use the official PDF.js build
+await definePDFJSModule(() => import('pdfjs-dist'))
+
+const pdf = await readFile('./dummy.pdf')
+const buffer = new Uint8Array(pdf)
+const pageNumber = 1
+
+const result = await renderPageAsImage(buffer, pageNumber, {
+  canvasImport: () => import('@napi-rs/canvas'),
+  scale: 2,
+})
+await writeFile('dummy-page-1.png', new Uint8Array(result))
+```
+
+```ts
+import { definePDFJSModule, renderPageAsImage } from 'unpdf'
+
+await definePDFJSModule(() => import('pdfjs-dist'))
+
+const pdf = await readFile('./dummy.pdf')
+const buffer = new Uint8Array(pdf)
+const pageNumber = 1
+
+const result = await renderPageAsImage(buffer, pageNumber, {
+  canvasImport: () => import('@napi-rs/canvas'),
+  scale: 2,
+  toDataURL: true,
+})
+
+const html = `<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Dummy Page</title>
+  </head>
+  <body>
+    <img alt="Example Page" src="${result}">
+  </body>
+</html>`
+
+await writeFile('dummy-page-1.html', html)
+```
+
+## License
+
+[MIT](./LICENSE) License © 2023-PRESENT [Johann Schopplich](https://github.com/johannschopplich)

package/node_modules/unpdf/dist/index.cjs

@@ -0,0 +1,340 @@
+'use strict';
+
+let resolvedModule;
+const isNode = globalThis.process?.release?.name === "node";
+const isBrowser = typeof window !== "undefined";
+async function getDocumentProxy(data, options = {}) {
+  const { getDocument } = await getResolvedPDFJS();
+  const pdf = await getDocument({
+    data,
+    isEvalSupported: false,
+    // See: https://github.com/mozilla/pdf.js/issues/4244#issuecomment-1479534301
+    useSystemFonts: true,
+    ...options
+  }).promise;
+  return pdf;
+}
+async function getResolvedPDFJS() {
+  if (!resolvedModule) {
+    await resolvePDFJSImport();
+  }
+  return resolvedModule;
+}
+async function resolvePDFJSImport(pdfjsResolver, { reload = false } = {}) {
+  if (resolvedModule && !reload) {
+    return;
+  }
+  if (pdfjsResolver) {
+    try {
+      resolvedModule = await interopDefault(pdfjsResolver());
+      return;
+    } catch (error) {
+      throw new Error(`PDF.js could not be resolved: ${error}`);
+    }
+  }
+  try {
+    resolvedModule = await import('unpdf/pdfjs');
+  } catch (error) {
+    throw new Error(`Serverless PDF.js bundle could not be resolved: ${error}`);
+  }
+}
+function isPDFDocumentProxy(data) {
+  return typeof data === "object" && data !== null && "_pdfInfo" in data;
+}
+async function interopDefault(m) {
+  const resolved = await m;
+  return resolved.default || resolved;
+}
+
+let resolvedCanvasModule;
+/**
+ * Derived from the PDF.js project by the Mozilla Foundation.
+ * @see https://github.com/mozilla/pdf.js/blob/b8de9a372f9bbf7e33adb362eeae5ef1919dba73/src/display/canvas_factory.js#L18
+ * @license Apache-2.0
+ */
+class BaseCanvasFactory {
+  #enableHWA = false;
+  constructor({ enableHWA = false } = {}) {
+    this.#enableHWA = enableHWA;
+  }
+  create(width, height) {
+    const canvas = this._createCanvas(width, height);
+    return {
+      canvas,
+      context: canvas.getContext("2d", {
+        willReadFrequently: !this.#enableHWA
+      })
+    };
+  }
+  reset({ canvas }, width, height) {
+    if (!canvas) {
+      throw new Error("Canvas is not specified");
+    }
+    canvas.width = width;
+    canvas.height = height;
+  }
+  destroy(context) {
+    if (!context.canvas) {
+      throw new Error("Canvas is not specified");
+    }
+    context.canvas.width = 0;
+    context.canvas.height = 0;
+    context.canvas = void 0;
+    context.context = void 0;
+  }
+  // eslint-disable-next-line unused-imports/no-unused-vars
+  _createCanvas(width, height) {
+    throw new Error("Not implemented");
+  }
+}
+/**
+ * Derived from the PDF.js project by the Mozilla Foundation.
+ * @see https://github.com/mozilla/pdf.js/blob/b8de9a372f9bbf7e33adb362eeae5ef1919dba73/src/display/canvas_factory.js#L18
+ * @license Apache-2.0
+ */
+class DOMCanvasFactory extends BaseCanvasFactory {
+  _document;
+  constructor({ ownerDocument = globalThis.document, enableHWA = false } = {}) {
+    super({ enableHWA });
+    this._document = ownerDocument;
+  }
+  _createCanvas(width, height) {
+    const canvas = this._document.createElement("canvas");
+    canvas.width = width;
+    canvas.height = height;
+    return canvas;
+  }
+}
+class NodeCanvasFactory extends BaseCanvasFactory {
+  constructor({ enableHWA = false } = {}) {
+    super({ enableHWA });
+  }
+  _createCanvas(width, height) {
+    if (!resolvedCanvasModule) {
+      throw new Error("@napi-rs/canvas module is not resolved");
+    }
+    return resolvedCanvasModule.createCanvas(width, height);
+  }
+}
+async function resolveCanvasModule(canvasImport) {
+  resolvedCanvasModule ??= await interopDefault(canvasImport());
+}
+function injectCanvasConstructors() {
+  if (!resolvedCanvasModule)
+    return;
+  if (typeof globalThis.DOMMatrix === "undefined")
+    globalThis.DOMMatrix = resolvedCanvasModule.DOMMatrix;
+  if (typeof globalThis.ImageData === "undefined")
+    globalThis.ImageData = resolvedCanvasModule.ImageData;
+  if (typeof globalThis.Path2D === "undefined")
+    globalThis.Path2D = resolvedCanvasModule.Path2D;
+}
+
+async function extractImages$1(data, pageNumber) {
+  const pdf = isPDFDocumentProxy(data) ? data : await getDocumentProxy(data);
+  if (pageNumber < 1 || pageNumber > pdf.numPages) {
+    throw new Error(`Invalid page number. Must be between 1 and ${pdf.numPages}.`);
+  }
+  const page = await pdf.getPage(pageNumber);
+  const operatorList = await page.getOperatorList();
+  const { OPS } = await getResolvedPDFJS();
+  const images = [];
+  for (let i = 0; i < operatorList.fnArray.length; i++) {
+    const op = operatorList.fnArray[i];
+    if (op !== OPS.paintImageXObject) {
+      continue;
+    }
+    const imageKey = operatorList.argsArray[i][0];
+    const image = await new Promise(
+      (resolve) => (imageKey.startsWith("g_") ? page.commonObjs : page.objs).get(imageKey, resolve)
+    );
+    if (!image || !image.data || !image.width || !image.height) {
+      continue;
+    }
+    const { width, height, data: data2 } = image;
+    const calculatedChannels = data2.length / (width * height);
+    if (![1, 3, 4].includes(calculatedChannels)) {
+      continue;
+    }
+    const channels = calculatedChannels;
+    images.push({
+      data: data2,
+      width,
+      height,
+      channels,
+      key: imageKey
+    });
+  }
+  return images;
+}
+async function renderPageAsImage$1(data, pageNumber, options = {}) {
+  const CanvasFactory = await createIsomorphicCanvasFactory(options.canvasImport);
+  const pdf = isPDFDocumentProxy(data) ? data : await getDocumentProxy(data, { CanvasFactory });
+  const page = await pdf.getPage(pageNumber);
+  if (pageNumber < 1 || pageNumber > pdf.numPages) {
+    throw new Error(`Invalid page number. Must be between 1 and ${pdf.numPages}.`);
+  }
+  const defaultViewport = page.getViewport({ scale: 1 });
+  let scale = options.scale || 1;
+  if (options.width) {
+    scale = options.width / defaultViewport.width;
+  } else if (options.height) {
+    scale = options.height / defaultViewport.height;
+  }
+  const viewport = page.getViewport({ scale: Math.max(0, scale) });
+  const drawingContext = new CanvasFactory().create(viewport.width, viewport.height);
+  await page.render({
+    canvas: drawingContext.canvas,
+    canvasContext: drawingContext.context,
+    viewport
+  }).promise;
+  const dataUrl = drawingContext.canvas.toDataURL();
+  if (options.toDataURL) {
+    return dataUrl;
+  }
+  const response = await fetch(dataUrl);
+  return await response.arrayBuffer();
+}
+async function createIsomorphicCanvasFactory(canvasImport) {
+  if (isBrowser)
+    return DOMCanvasFactory;
+  if (isNode) {
+    if (!canvasImport) {
+      throw new Error('Parameter "canvasImport" is required in Node.js environment.');
+    }
+    await resolveCanvasModule(canvasImport);
+    injectCanvasConstructors();
+    return NodeCanvasFactory;
+  }
+  throw new Error("Unsupported environment for canvas creation.");
+}
+
+async function extractLinks$1(data) {
+  const pdf = isPDFDocumentProxy(data) ? data : await getDocumentProxy(data);
+  const pageLinks = await Promise.all(
+    Array.from({ length: pdf.numPages }, (_, i) => getPageLinks(pdf, i + 1))
+  );
+  return {
+    totalPages: pdf.numPages,
+    links: pageLinks.flat()
+  };
+}
+async function getPageLinks(document, pageNumber) {
+  const page = await document.getPage(pageNumber);
+  const annotations = await page.getAnnotations();
+  const links = [];
+  for (const annotation of annotations) {
+    if (annotation.subtype === "Link" && annotation.url) {
+      links.push(annotation.url);
+    }
+  }
+  return links;
+}
+
+const XMP_DATE_PROPERTIES = [
+  "xmp:createdate",
+  "xmp:modifydate",
+  "xmp:metadatadate",
+  "xap:createdate",
+  "xap:modifydate",
+  "xap:metadatadate"
+];
+async function getMeta$1(data, options = {}) {
+  const pdf = isPDFDocumentProxy(data) ? data : await getDocumentProxy(data);
+  const meta = await pdf.getMetadata();
+  const info = meta?.info || {};
+  if (options.parseDates) {
+    const { PDFDateString } = await getResolvedPDFJS();
+    if (info?.CreationDate) {
+      info.CreationDate = PDFDateString.toDateObject(info?.CreationDate);
+    }
+    if (info?.ModDate) {
+      info.ModDate = PDFDateString.toDateObject(info?.ModDate);
+    }
+    if (meta.metadata) {
+      meta.metadata = new Proxy(meta.metadata, {
+        get(target, prop) {
+          if (prop === "get") {
+            return (name) => {
+              const value = target.get(name);
+              if (XMP_DATE_PROPERTIES.includes(name) && value) {
+                return parseISODateString(value);
+              }
+              return value;
+            };
+          }
+          return target[prop];
+        }
+      });
+    }
+  }
+  return {
+    info,
+    metadata: meta?.metadata || {}
+  };
+}
+function parseISODateString(isoDateString) {
+  if (!isoDateString)
+    return;
+  const parsedDate = Date.parse(isoDateString);
+  if (!Number.isNaN(parsedDate)) {
+    return new Date(parsedDate);
+  }
+}
+
+async function extractText$1(data, options = {}) {
+  const { mergePages = false } = options;
+  const pdf = isPDFDocumentProxy(data) ? data : await getDocumentProxy(data);
+  const texts = await Promise.all(
+    Array.from({ length: pdf.numPages }, (_, i) => getPageText(pdf, i + 1))
+  );
+  return {
+    totalPages: pdf.numPages,
+    text: mergePages ? texts.join("\n").replace(/\s+/g, " ") : texts
+  };
+}
+async function getPageText(document, pageNumber) {
+  const page = await document.getPage(pageNumber);
+  const content = await page.getTextContent();
+  return content.items.filter((item) => item.str != null).map((item) => item.str + (item.hasEOL ? "\n" : "")).join("");
+}
+
+async function definePDFJSModule(pdfjs) {
+  await resolvePDFJSImport(pdfjs, { reload: true });
+}
+async function configureUnPDF(options) {
+  await resolvePDFJSImport(options.pdfjs, { reload: true });
+}
+
+const getMeta = async (...args) => {
+  await resolvePDFJSImport();
+  return await getMeta$1(...args);
+};
+const extractText = async (...args) => {
+  await resolvePDFJSImport();
+  return await extractText$1(...args);
+};
+const extractImages = async (...args) => {
+  await resolvePDFJSImport();
+  return await extractImages$1(...args);
+};
+const renderPageAsImage = async (...args) => {
+  await resolvePDFJSImport();
+  return await renderPageAsImage$1(...args);
+};
+const extractLinks = async (...args) => {
+  await resolvePDFJSImport();
+  return await extractLinks$1(...args);
+};
+
+exports.configureUnPDF = configureUnPDF;
+exports.createIsomorphicCanvasFactory = createIsomorphicCanvasFactory;
+exports.definePDFJSModule = definePDFJSModule;
+exports.extractImages = extractImages;
+exports.extractLinks = extractLinks;
+exports.extractText = extractText;
+exports.getDocumentProxy = getDocumentProxy;
+exports.getMeta = getMeta;
+exports.getResolvedPDFJS = getResolvedPDFJS;
+exports.renderPageAsImage = renderPageAsImage;
+exports.resolvePDFJSImport = resolvePDFJSImport;