@heripo/pdf-parser 0.1.8 → 0.1.9

package/README.ko.md

@@ -3,7 +3,7 @@
 > PDF 파싱 라이브러리 - Docling SDK를 활용한 OCR 지원
 
 [![npm version](https://img.shields.io/npm/v/@heripo/pdf-parser.svg)](https://www.npmjs.com/package/@heripo/pdf-parser)
-[![Node.js](https://img.shields.io/badge/Node.js-%3E%3D22-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
+[![Node.js](https://img.shields.io/badge/Node.js-%3E%3D24-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
 [![Python](https://img.shields.io/badge/Python-3.9--3.12-3776AB?logo=python&logoColor=white)](https://www.python.org/)
 ![coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](../../LICENSE)
@@ -46,7 +46,7 @@
 
 ### 필수 의존성
 
-#### 1. Node.js >= 22.0.0
+#### 1. Node.js >= 24.0.0
 
 ```bash
 brew install node
@@ -72,7 +72,7 @@ python3.11 --version
 
 #### 4. poppler (PDF 텍스트 추출)
 
-OCR 전략 시스템의 텍스트 레이어 사전 검사(`pdftotext`)에 필요합니다.
+PDF 페이지 수 확인(`pdfinfo`)과 텍스트 레이어 추출(`pdftotext`)에 필요하며, OCR 전략 시스템의 텍스트 레이어 사전 검사에 사용됩니다.
 
 ```bash
 brew install poppler
@@ -281,12 +281,12 @@ const outputPath = await pdfParser.parse(
 
 `@heripo/pdf-parser`는 다음 시스템 레벨 의존성이 필요합니다:
 
-| 의존성  | 필수 버전  | 설치 방법                  | 용도                                      |
-| ------- | ---------- | -------------------------- | ----------------------------------------- |
-| Python  | 3.9 - 3.12 | `brew install python@3.11` | Docling SDK 실행 환경                     |
-| poppler | Any        | `brew install poppler`     | OCR 전략용 텍스트 레이어 추출 (pdftotext) |
-| jq      | Any        | `brew install jq`          | JSON 처리 (변환 결과 파싱)                |
-| lsof    | Any        | macOS 기본 설치됨          | docling-serve 포트 관리                   |
+| 의존성  | 필수 버전  | 설치 방법                  | 용도                                                           |
+| ------- | ---------- | -------------------------- | -------------------------------------------------------------- |
+| Python  | 3.9 - 3.12 | `brew install python@3.11` | Docling SDK 실행 환경                                          |
+| poppler | Any        | `brew install poppler`     | PDF 페이지 수 확인 (pdfinfo) 및 텍스트 레이어 추출 (pdftotext) |
+| jq      | Any        | `brew install jq`          | JSON 처리 (변환 결과 파싱)                                     |
+| lsof    | Any        | macOS 기본 설치됨          | docling-serve 포트 관리                                        |
 
 > ⚠️ **Python 3.13+는 지원하지 않습니다.** Docling SDK의 일부 의존성이 Python 3.13과 호환되지 않습니다.
 
@@ -411,6 +411,16 @@ const pdfParser = new PDFParser({
 brew install jq
 ```
 
+### poppler를 찾을 수 없음
+
+**증상**: `poppler is not installed. Please install poppler using: brew install poppler`
+
+**해결**:
+
+```bash
+brew install poppler
+```
+
 ### 포트 충돌
 
 **증상**: `Port 5001 is already in use`

package/README.md

@@ -3,7 +3,7 @@
 > PDF parsing library - OCR support with Docling SDK
 
 [![npm version](https://img.shields.io/npm/v/@heripo/pdf-parser.svg)](https://www.npmjs.com/package/@heripo/pdf-parser)
-[![Node.js](https://img.shields.io/badge/Node.js-%3E%3D22-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
+[![Node.js](https://img.shields.io/badge/Node.js-%3E%3D24-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
 [![Python](https://img.shields.io/badge/Python-3.9--3.12-3776AB?logo=python&logoColor=white)](https://www.python.org/)
 ![coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](../../LICENSE)
@@ -46,7 +46,7 @@
 
 ### Required Dependencies
 
-#### 1. Node.js >= 22.0.0
+#### 1. Node.js >= 24.0.0
 
 ```bash
 brew install node
@@ -72,7 +72,7 @@ python3.11 --version
 
 #### 4. poppler (PDF text extraction)
 
-Required for the OCR strategy system's text layer pre-check (`pdftotext`).
+Required for PDF page counting (`pdfinfo`) and text layer extraction (`pdftotext`), used by the OCR strategy system's text layer pre-check.
 
 ```bash
 brew install poppler
@@ -281,12 +281,12 @@ Archaeological excavation report PDFs have the following characteristics:
 
 `@heripo/pdf-parser` requires the following system-level dependencies:
 
-| Dependency | Required Version | Installation               | Purpose                                            |
-| ---------- | ---------------- | -------------------------- | -------------------------------------------------- |
-| Python     | 3.9 - 3.12       | `brew install python@3.11` | Docling SDK runtime                                |
-| poppler    | Any              | `brew install poppler`     | Text layer extraction for OCR strategy (pdftotext) |
-| jq         | Any              | `brew install jq`          | JSON processing (conversion result parsing)        |
-| lsof       | Any              | Included with macOS        | docling-serve port management                      |
+| Dependency | Required Version | Installation               | Purpose                                                           |
+| ---------- | ---------------- | -------------------------- | ----------------------------------------------------------------- |
+| Python     | 3.9 - 3.12       | `brew install python@3.11` | Docling SDK runtime                                               |
+| poppler    | Any              | `brew install poppler`     | PDF page counting (pdfinfo) and text layer extraction (pdftotext) |
+| jq         | Any              | `brew install jq`          | JSON processing (conversion result parsing)                       |
+| lsof       | Any              | Included with macOS        | docling-serve port management                                     |
 
 > ⚠️ **Python 3.13+ is not supported.** Some Docling SDK dependencies are not compatible with Python 3.13.
 
@@ -411,6 +411,16 @@ const pdfParser = new PDFParser({
 brew install jq
 ```
 
+### poppler Not Found
+
+**Symptom**: `poppler is not installed. Please install poppler using: brew install poppler`
+
+**Solution**:
+
+```bash
+brew install poppler
+```
+
 ### Port Conflict
 
 **Symptom**: `Port 5001 is already in use`

package/dist/index.cjs

@@ -1060,28 +1060,28 @@ var ImageExtractor = class _ImageExtractor {
     const baseName = filename.replace((0, import_node_path2.extname)(filename), "");
     const jsonPath = (0, import_node_path2.join)(outputDir, `${baseName}.json`);
     try {
-      const pagesDir = (0, import_node_path2.join)(outputDir, "pages");
-      if (!(0, import_node_fs.existsSync)(pagesDir)) {
-        (0, import_node_fs.mkdirSync)(pagesDir, { recursive: true });
+      const imagesDir = (0, import_node_path2.join)(outputDir, "images");
+      if (!(0, import_node_fs.existsSync)(imagesDir)) {
+        (0, import_node_fs.mkdirSync)(imagesDir, { recursive: true });
       }
       const base64Images = await _ImageExtractor.extractBase64ImagesFromJsonWithJq(jsonSourcePath);
       base64Images.forEach((base64Data, index) => {
         _ImageExtractor.extractBase64ImageToFile(
           base64Data,
-          pagesDir,
+          imagesDir,
           index,
-          "page",
-          "pages"
+          "pic",
+          "images"
         );
       });
       logger.info(
-        `[PDFConverter] Extracted ${base64Images.length} images from JSON to ${pagesDir}`
+        `[PDFConverter] Extracted ${base64Images.length} picture images from JSON to ${imagesDir}`
       );
       const replacedCount = await _ImageExtractor.replaceBase64ImagesInJsonWithJq(
         jsonSourcePath,
         jsonPath,
-        "pages",
-        "page"
+        "images",
+        "pic"
       );
       logger.info(
         `[PDFConverter] Replaced ${replacedCount} base64 images with file paths`
@@ -1817,6 +1817,7 @@ var VlmTextCorrector = class {
 };
 
 // src/samplers/ocr-strategy-sampler.ts
+var import_model = require("@heripo/model");
 var import_node_fs4 = require("fs");
 var import_v42 = require("zod/v4");
 var SAMPLE_DPI = 150;
@@ -1889,7 +1890,7 @@ var OcrStrategySampler = class {
       `[OcrStrategySampler] Sampling ${sampleIndices.length} of ${renderResult.pageCount} pages: [${sampleIndices.map((i) => i + 1).join(", ")}]`
     );
     let sampledCount = 0;
-    let detectedLanguages;
+    const languageFrequency = /* @__PURE__ */ new Map();
     for (const idx of sampleIndices) {
       sampledCount++;
       const pageFile = renderResult.pageFiles[idx];
@@ -1899,14 +1900,17 @@ var OcrStrategySampler = class {
         model,
         options
       );
-      detectedLanguages = pageAnalysis.detectedLanguages;
+      for (const lang of pageAnalysis.detectedLanguages) {
+        languageFrequency.set(lang, (languageFrequency.get(lang) ?? 0) + 1);
+      }
       if (pageAnalysis.hasKoreanHanjaMix) {
         this.logger.info(
           `[OcrStrategySampler] Korean-Hanja mix detected on page ${idx + 1} \u2192 VLM strategy`
         );
+        const detectedLanguages2 = this.aggregateLanguages(languageFrequency);
         return {
           method: "vlm",
-          detectedLanguages,
+          detectedLanguages: detectedLanguages2,
           reason: `Korean-Hanja mix detected on page ${idx + 1}`,
           sampledPages: sampledCount,
           totalPages: renderResult.pageCount
@@ -1916,6 +1920,7 @@ var OcrStrategySampler = class {
     this.logger.info(
       "[OcrStrategySampler] No Korean-Hanja mix detected \u2192 ocrmac strategy"
     );
+    const detectedLanguages = this.aggregateLanguages(languageFrequency);
     return {
       method: "ocrmac",
       detectedLanguages,
@@ -2020,8 +2025,9 @@ var OcrStrategySampler = class {
   }
   /**
    * Analyze a single sample page for Korean-Hanja mixed script and primary language.
+   * Normalizes raw VLM language responses to valid BCP 47 tags, filtering out invalid ones.
    *
-   * @returns Object with Korean-Hanja detection result and detected languages
+   * @returns Object with Korean-Hanja detection result and normalized detected languages
    */
   async analyzeSamplePage(pageFile, pageNo, model, options) {
     this.logger.debug(
@@ -2055,14 +2061,23 @@ var OcrStrategySampler = class {
       options.aggregator.track(result.usage);
     }
     const output = result.output;
+    const normalizedLanguages = output.detectedLanguages.map(import_model.normalizeToBcp47).filter((tag) => tag !== null);
     this.logger.debug(
-      `[OcrStrategySampler] Page ${pageNo}: hasKoreanHanjaMix=${output.hasKoreanHanjaMix}, detectedLanguages=${output.detectedLanguages.join(",")}`
+      `[OcrStrategySampler] Page ${pageNo}: hasKoreanHanjaMix=${output.hasKoreanHanjaMix}, detectedLanguages=${normalizedLanguages.join(",")}`
     );
     return {
       hasKoreanHanjaMix: output.hasKoreanHanjaMix,
-      detectedLanguages: output.detectedLanguages
+      detectedLanguages: normalizedLanguages
     };
   }
+  /**
+   * Aggregate language frequency map into a sorted array.
+   * Returns languages sorted by frequency (descending), or undefined if empty.
+   */
+  aggregateLanguages(frequencyMap) {
+    if (frequencyMap.size === 0) return void 0;
+    return [...frequencyMap.entries()].sort((a, b) => b[1] - a[1]).map(([lang]) => lang);
+  }
 };
 
 // src/utils/local-file-server.ts
@@ -2531,6 +2546,7 @@ var PDFConverter = class {
     const outputDir = (0, import_node_path7.join)(cwd, "output", reportId);
     try {
       await this.processConvertedFiles(zipPath, extractDir, outputDir);
+      await this.renderPageImages(url, outputDir);
       if (abortSignal?.aborted) {
         this.logger.info("[PDFConverter] Conversion aborted before callback");
         const error = new Error("PDF conversion was aborted");
@@ -2586,6 +2602,8 @@ var PDFConverter = class {
         framework: "livetext"
       },
       generate_picture_images: true,
+      generate_page_images: false,
+      // Page images are rendered by PageRenderer (ImageMagick) after conversion
       images_scale: 2,
       /**
        * While disabling this option yields the most accurate text extraction for readable PDFs,
@@ -2734,6 +2752,40 @@ var PDFConverter = class {
       outputDir
     );
   }
+  /**
+   * Render page images from the source PDF using ImageMagick and update result.json.
+   * Replaces Docling's generate_page_images which fails on large PDFs
+   * due to memory limits when embedding all page images as base64.
+   */
+  async renderPageImages(url, outputDir) {
+    if (!url.startsWith("file://")) {
+      this.logger.warn(
+        "[PDFConverter] Page image rendering skipped: only supported for local files (file:// URLs)"
+      );
+      return;
+    }
+    const pdfPath = url.slice(7);
+    this.logger.info(
+      "[PDFConverter] Rendering page images with ImageMagick..."
+    );
+    const renderer = new PageRenderer(this.logger);
+    const renderResult = await renderer.renderPages(pdfPath, outputDir);
+    const resultPath = (0, import_node_path7.join)(outputDir, "result.json");
+    const doc = JSON.parse((0, import_node_fs7.readFileSync)(resultPath, "utf-8"));
+    for (const page of Object.values(doc.pages)) {
+      const pageNo = page.page_no;
+      const fileIndex = pageNo - 1;
+      if (fileIndex >= 0 && fileIndex < renderResult.pageCount) {
+        page.image.uri = `pages/page_${fileIndex}.png`;
+        page.image.mimetype = "image/png";
+        page.image.dpi = 300;
+      }
+    }
+    await (0, import_promises.writeFile)(resultPath, JSON.stringify(doc, null, 2));
+    this.logger.info(
+      `[PDFConverter] Rendered ${renderResult.pageCount} page images`
+    );
+  }
 };
 
 // src/core/pdf-parser.ts
@@ -2772,6 +2824,7 @@ var PDFParser = class {
     this.logger.info("[PDFParser] Initializing...");
     this.checkOperatingSystem();
     this.checkJqInstalled();
+    this.checkPopplerInstalled();
     this.checkMacOSVersion();
     if (this.enableImagePdfFallback && !this.baseUrl) {
       this.checkImageMagickInstalled();
@@ -2828,6 +2881,15 @@ var PDFParser = class {
       );
     }
   }
+  checkPopplerInstalled() {
+    try {
+      (0, import_node_child_process3.execSync)("which pdftotext", { stdio: "ignore" });
+    } catch {
+      throw new Error(
+        "poppler is not installed. Please install poppler using: brew install poppler"
+      );
+    }
+  }
   checkMacOSVersion() {
     try {
       const versionOutput = (0, import_node_child_process3.execSync)("sw_vers -productVersion", {