@heripo/pdf-parser 0.1.27 → 0.1.28

package/README.ko.md

@@ -35,7 +35,7 @@
 ## 주요 기능
 
 - **고품질 OCR**: Docling SDK를 활용한 문서 인식 (ocrmac / Apple Vision Framework)
-- **혼합 문자 자동 감지 및 보정**: 한글·한자 혼용 페이지를 자동 감지하여 VLM으로 보정
+- **한국어 보고서 VLM 보정**: 한국어 보고서를 자동 감지하고 모든 페이지에 VLM 텍스트 보정을 적용
 - **Apple Silicon 최적화**: M1/M2/M3/M4/M5 칩에서 GPU 가속 지원
 - **자동 환경 설정**: Python 가상환경 및 docling-serve 자동 설치
 - **이미지 추출**: PDF 내 이미지 자동 추출 및 저장
@@ -233,16 +233,16 @@ await pdfParser.dispose();
 
 **ocrmac(Apple Vision Framework)은 매우 우수한 OCR 엔진입니다** -- 무료이고, GPU 가속을 지원하며, 고품질 결과를 제공합니다. 수천~수백만 권의 고고학 보고서를 처리할 때 이만한 솔루션이 없습니다.
 
-**그러나 ocrmac은 혼합 문자 체계를 처리하지 못합니다.** 한글·한자 혼용 문서(그리고 잠재적으로 다른 혼합 문자 조합)에서 보조 문자가 깨진 텍스트로 변환됩니다. 전체 파이프라인을 비용이 높은 VLM으로 전환하는 대신, **문제가 있는 페이지만 타겟팅**하여 VLM으로 보정합니다.
+**그러나 한국어 고고학 보고서는 문자 체계 인식 보정이 자주 필요합니다.** 한자 복원, CJK mojibake, 음독 치환, 기관명 등은 표준 OCR만으로 안정적으로 처리하기 어렵습니다. VLM을 기본 OCR 엔진으로 쓰는 대신, 빠른 ocrmac 파이프라인을 먼저 실행한 뒤 한국어 보고서에 VLM 텍스트 보정을 적용합니다.
 
-### 2단계 감지 (`OcrStrategySampler`)
+### 2단계 한국어 감지 (`OcrStrategySampler`)
 
-1. **텍스트 레이어 사전 검사** (비용 없음): `pdftotext`로 문서의 텍스트 레이어를 추출하여 한글과 CJK 문자가 모두 존재하는지 확인합니다. 둘 다 존재하면 즉시 혼합 문자 문서로 판별합니다.
-2. **VLM 샘플링** (필요 시에만): 최대 15페이지를 샘플링(앞뒤 10%는 표지·부록으로 제외)하여 Vision LLM으로 분석합니다. 최초 한글·한자 혼용 감지 시 즉시 종료하여 API 비용을 최소화합니다.
+1. **텍스트 레이어 사전 검사** (비용 없음): `pdftotext`로 문서의 텍스트 레이어를 추출하여 한글을 확인합니다. 한글이 있으면 즉시 한국어 문서로 판별합니다.
+2. **VLM 샘플링** (필요 시에만): 최대 15페이지를 샘플링(앞뒤 10%는 표지·부록으로 제외)하여 Vision LLM으로 언어를 판정합니다. `ko-KR` 감지 시 즉시 종료합니다.
 
-### 페이지별 보정 (`VlmTextCorrector`)
+### 전체 문서 보정 (`VlmTextCorrector`)
 
-혼합 문자 페이지가 감지되면, 해당 페이지만 VLM에 전송하여 보정합니다:
+한국어 보고서가 감지되면, 모든 페이지를 VLM에 전송하여 보정합니다:
 
 - 각 페이지의 OCR 텍스트 요소와 표 셀을 추출
 - `pdftotext` 참조 텍스트를 품질 기준으로 활용
@@ -261,7 +261,7 @@ const tokenUsageReport = await pdfParser.parse(
     // OCR 전략 샘플링 활성화 (Vision LLM 모델 제공)
     strategySamplerModel: openai('gpt-5.1'),
 
-    // VLM 텍스트 보정 모델 (혼합 문자 감지 시 필요)
+    // VLM 텍스트 보정 모델 (한국어 보고서 감지 시 필요)
     vlmProcessorModel: openai('gpt-5.1'),
 
     // VLM 페이지 처리 동시성 (기본값: 1)

package/README.md

@@ -35,7 +35,7 @@
 ## Key Features
 
 - **High-Quality OCR**: Document recognition using Docling SDK (ocrmac / Apple Vision Framework)
-- **Mixed Script Auto-Detection & Correction**: Automatically detects Korean-Hanja mixed pages and corrects them via VLM
+- **Korean Report VLM Correction**: Automatically detects Korean reports and applies VLM text correction to all pages
 - **Apple Silicon Optimized**: GPU acceleration on M1/M2/M3/M4/M5 chips
 - **Automatic Environment Setup**: Automatic Python virtual environment and docling-serve installation
 - **Image Extraction**: Automatic extraction and saving of images from PDFs
@@ -233,16 +233,16 @@ await pdfParser.dispose();
 
 **ocrmac (Apple Vision Framework) is an excellent OCR engine** -- it's free, GPU-accelerated, and delivers high-quality results. For processing thousands to millions of archaeological reports, there's no better solution.
 
-**However, ocrmac cannot handle mixed character systems.** Documents containing Korean-Hanja combinations (and potentially other mixed scripts) produce garbled text for the non-primary script. Rather than switching the entire pipeline to a costly VLM, the system **targets only the affected pages** for VLM correction, minimizing cost and processing time.
+**However, Korean archaeological reports often need script-aware correction.** Hanja restoration, CJK mojibake, phonetic substitutions, and institution names can be unreliable with standard OCR alone. Rather than using VLM as the primary OCR engine, the system runs the fast ocrmac pipeline first and then applies VLM text correction to Korean reports.
 
-### Two-Stage Detection (`OcrStrategySampler`)
+### Two-Stage Korean Detection (`OcrStrategySampler`)
 
-1. **Text Layer Pre-Check** (zero cost): Extracts the document's text layer using `pdftotext` and checks for both Hangul and CJK characters. If both are present, the document is immediately flagged as mixed-script.
-2. **VLM Sampling** (only when needed): Samples up to 15 pages (trimming 10% from front/back to skip covers and appendices) and analyzes them with a Vision LLM. Uses early exit on first Korean-Hanja mix detection to minimize API costs.
+1. **Text Layer Pre-Check** (zero cost): Extracts the document's text layer using `pdftotext` and checks for Hangul. If Hangul is present, the document is immediately treated as Korean.
+2. **VLM Sampling** (only when needed): Samples up to 15 pages (trimming 10% from front/back to skip covers and appendices) and analyzes them with a Vision LLM. Uses early exit when `ko-KR` is detected.
 
-### Per-Page Correction (`VlmTextCorrector`)
+### Full-Document Correction (`VlmTextCorrector`)
 
-When mixed-script pages are detected, only those pages are sent to the VLM for correction:
+When a Korean report is detected, every page is sent to the VLM for correction:
 
 - Extracts OCR text elements and table cells from each page
 - Uses `pdftotext` reference text as a quality anchor
@@ -261,7 +261,7 @@ const tokenUsageReport = await pdfParser.parse(
     // Enable OCR strategy sampling (provide a Vision LLM model)
     strategySamplerModel: openai('gpt-5.1'),
 
-    // VLM model for text correction (required when mixed-script is detected)
+    // VLM model for text correction (required when Korean reports are detected)
     vlmProcessorModel: openai('gpt-5.1'),
 
     // Concurrency for VLM page processing (default: 1)

package/dist/index.cjs

@@ -6545,15 +6545,12 @@ var import_model = require("@heripo/model");
 var import_node_fs12 = require("fs");
 var import_v42 = require("zod/v4");
 
-// src/prompts/korean-hanja-mix-prompt.ts
-var KOREAN_HANJA_MIX_PROMPT = `Look at this page image carefully. Does it contain any Hanja (\u6F22\u5B57/Chinese characters) mixed with Korean text?
+// src/prompts/korean-document-detection-prompt.ts
+var KOREAN_DOCUMENT_DETECTION_PROMPT = `Look at this page image carefully and identify all languages present on this page.
 
-Hanja examples: \u907A\u8E5F, \u767C\u6398, \u8ABF\u67FB, \u5831\u544A\u66F8, \u6587\u5316\u8CA1
-Note: Hanja are Chinese characters used in Korean documents, different from modern Korean (\uD55C\uAE00).
+Return an array of ocrmac-compatible language tags ordered by prevalence (primary language first).
+Use ko-KR when Korean text is present, even when the page also contains Hanja (\u6F22\u5B57/Chinese characters), English, numbers, tables, or captions.
 
-Answer whether any Hanja characters are present on this page.
-
-Also identify all languages present on this page. Return an array of ocrmac-compatible language tags ordered by prevalence (primary language first).
 Supported tags: ar-SA, ars-SA, cs-CZ, da-DK, de-DE, en-US, es-ES, fr-FR, id-ID, it-IT, ja-JP, ko-KR, ms-MY, nb-NO, nl-NL, nn-NO, no-NO, pl-PL, pt-BR, ro-RO, ru-RU, sv-SE, th-TH, tr-TR, uk-UA, vi-VT, yue-Hans, yue-Hant, zh-Hans, zh-Hant.
 Examples: ["ko-KR", "en-US"], ["ja-JP"], ["zh-Hant", "en-US"]`;
 
@@ -6561,12 +6558,8 @@ Examples: ["ko-KR", "en-US"], ["ja-JP"], ["zh-Hant", "en-US"]`;
 var EDGE_TRIM_RATIO = 0.1;
 var DEFAULT_MAX_SAMPLE_PAGES = 15;
 var DEFAULT_MAX_RETRIES = 3;
-var CJK_REGEX = /[\u4E00-\u9FFF]/;
 var HANGUL_REGEX = /[\uAC00-\uD7AF]/;
-var koreanHanjaMixSchema = import_v42.z.object({
-  hasKoreanHanjaMix: import_v42.z.boolean().describe(
-    "Whether the page contains any Hanja (\u6F22\u5B57/Chinese characters) mixed with Korean text"
-  ),
+var koreanDocumentDetectionSchema = import_v42.z.object({
   detectedLanguages: import_v42.z.array(import_v42.z.string()).describe(
     'BCP 47 language tags of languages found on this page, ordered by prevalence (e.g., ["ko-KR", "en-US"])'
   )
@@ -6585,14 +6578,14 @@ var OcrStrategySampler = class {
    *
    * @param pdfPath - Path to the PDF file
    * @param outputDir - Directory for temporary rendered pages
-   * @param model - Vision language model for Korean-Hanja mix detection
+   * @param model - Vision language model for Korean document detection
    * @param options - Sampling options
    * @returns OcrStrategy with method ('ocrmac' or 'vlm') and metadata
    */
   async sample(pdfPath, outputDir, model, options) {
     const maxSamplePages = options?.maxSamplePages ?? DEFAULT_MAX_SAMPLE_PAGES;
     this.logger.info("[OcrStrategySampler] Starting OCR strategy sampling...");
-    const preCheckResult = await this.preCheckHanjaFromTextLayer(pdfPath);
+    const preCheckResult = await this.preCheckKoreanFromTextLayer(pdfPath);
     if (preCheckResult) {
       return preCheckResult;
     }
@@ -6623,9 +6616,9 @@ var OcrStrategySampler = class {
       model,
       options
     );
-    if (sampleResult.foundMix) {
+    if (sampleResult.foundKorean) {
       this.logger.info(
-        `[OcrStrategySampler] Korean-Hanja mix detected on page ${sampleResult.mixPageNo} \u2192 VLM strategy`
+        `[OcrStrategySampler] Korean document detected on page ${sampleResult.koreanPageNo} \u2192 VLM strategy`
       );
       const detectedLanguages2 = this.aggregateLanguages(
         sampleResult.languageFrequency
@@ -6633,13 +6626,13 @@ var OcrStrategySampler = class {
       return {
         method: "vlm",
         detectedLanguages: detectedLanguages2,
-        reason: `Korean-Hanja mix detected on page ${sampleResult.mixPageNo}`,
+        reason: `Korean document detected on page ${sampleResult.koreanPageNo}`,
         sampledPages: sampleResult.sampledCount,
         totalPages: renderResult.pageCount
       };
     }
     this.logger.info(
-      "[OcrStrategySampler] No Korean-Hanja mix detected \u2192 ocrmac strategy"
+      "[OcrStrategySampler] No Korean language detected \u2192 ocrmac strategy"
     );
     const detectedLanguages = this.aggregateLanguages(
       sampleResult.languageFrequency
@@ -6647,13 +6640,13 @@ var OcrStrategySampler = class {
     return {
       method: "ocrmac",
       detectedLanguages,
-      reason: `No Korean-Hanja mix detected in ${sampleResult.sampledCount} sampled pages`,
+      reason: `No Korean language detected in ${sampleResult.sampledCount} sampled pages`,
       sampledPages: sampleResult.sampledCount,
       totalPages: renderResult.pageCount
     };
   }
   /**
-   * Sample pages for Korean-Hanja mix detection with early exit.
+   * Sample pages for Korean language detection with early exit.
    */
   async samplePages(sampleIndices, pageFiles, model, options) {
     const languageFrequency = /* @__PURE__ */ new Map();
@@ -6669,72 +6662,52 @@ var OcrStrategySampler = class {
       for (const lang of pageAnalysis.detectedLanguages) {
         languageFrequency.set(lang, (languageFrequency.get(lang) ?? 0) + 1);
       }
-      if (pageAnalysis.hasKoreanHanjaMix) {
+      if (pageAnalysis.detectedLanguages.includes("ko-KR")) {
         return {
-          foundMix: true,
-          mixPageNo: idx + 1,
+          foundKorean: true,
+          koreanPageNo: idx + 1,
           sampledCount: i + 1,
           languageFrequency
         };
       }
     }
     return {
-      foundMix: false,
+      foundKorean: false,
       sampledCount: sampleIndices.length,
       languageFrequency
     };
   }
   /**
-   * Pre-check for Hangul-Hanja mix in PDF text layer using pdftotext.
-   * Extracts full document text in a single process and checks at document level.
-   * Only makes a definitive decision for Korean (Hangul) documents:
-   * - Hangul + Hanja (anywhere in document) → VLM (confirmed Korean-Hanja mix)
-   * - Hangul only → ocrmac with ko-KR (confirmed Korean)
+   * Pre-check for Korean text in PDF text layer using pdftotext.
+   * Extracts full document text in a single process and checks at document level:
+   * - Hangul anywhere in document → VLM (confirmed Korean document)
    * - No Hangul (English, Japanese, etc.) → null (delegates to VLM for language detection)
    */
-  async preCheckHanjaFromTextLayer(pdfPath) {
+  async preCheckKoreanFromTextLayer(pdfPath) {
     try {
       const totalPages = await this.textExtractor.getPageCount(pdfPath);
       if (totalPages === 0) return null;
       const fullText = await this.textExtractor.extractFullText(pdfPath);
       if (fullText.trim().length === 0) {
         this.logger.debug(
-          "[OcrStrategySampler] No Hangul in text layer, falling back to VLM sampling"
+          "[OcrStrategySampler] No text in text layer, falling back to VLM sampling"
         );
         return null;
       }
       const hasHangul = HANGUL_REGEX.test(fullText);
-      const hasHanja = CJK_REGEX.test(fullText);
       if (!hasHangul) {
         this.logger.debug(
           "[OcrStrategySampler] No Hangul in text layer, falling back to VLM sampling"
         );
         return null;
       }
-      if (hasHanja) {
-        const pageTextArray = fullText.split("\f");
-        const koreanHanjaMixPages = pageTextArray.flatMap(
-          (text, i) => CJK_REGEX.test(text) ? [i + 1] : []
-        );
-        this.logger.info(
-          `[OcrStrategySampler] Hangul-Hanja mix detected in text layer \u2192 VLM strategy (${koreanHanjaMixPages.length} pages with Hanja)`
-        );
-        return {
-          method: "vlm",
-          detectedLanguages: ["ko-KR"],
-          reason: "Hangul-Hanja mix found in PDF text layer",
-          koreanHanjaMixPages,
-          sampledPages: totalPages,
-          totalPages
-        };
-      }
       this.logger.info(
-        "[OcrStrategySampler] No Hangul-Hanja mix in text layer \u2192 ocrmac strategy"
+        "[OcrStrategySampler] Korean text detected in text layer \u2192 VLM strategy"
       );
       return {
-        method: "ocrmac",
+        method: "vlm",
         detectedLanguages: ["ko-KR"],
-        reason: `No Hangul-Hanja mix in PDF text layer (${totalPages} pages checked)`,
+        reason: `Korean text found in PDF text layer (${totalPages} pages checked)`,
         sampledPages: totalPages,
         totalPages
       };
@@ -6775,21 +6748,24 @@ var OcrStrategySampler = class {
     );
   }
   /**
-   * Analyze a single sample page for Korean-Hanja mixed script and primary language.
+   * Analyze a single sample page for language.
    * Normalizes raw VLM language responses to valid BCP 47 tags, filtering out invalid ones.
    *
-   * @returns Object with Korean-Hanja detection result and normalized detected languages
+   * @returns Object with normalized detected languages
    */
   async analyzeSamplePage(pageFile, pageNo, model, options) {
     this.logger.debug(
-      `[OcrStrategySampler] Analyzing page ${pageNo} for Korean-Hanja mix and language...`
+      `[OcrStrategySampler] Analyzing page ${pageNo} for language...`
     );
     const imageData = new Uint8Array((0, import_node_fs12.readFileSync)(pageFile));
     const messages = [
       {
         role: "user",
         content: [
-          { type: "text", text: KOREAN_HANJA_MIX_PROMPT },
+          {
+            type: "text",
+            text: KOREAN_DOCUMENT_DETECTION_PROMPT
+          },
           {
             type: "image",
             image: imageData,
@@ -6799,7 +6775,7 @@ var OcrStrategySampler = class {
       }
     ];
     const result = await LLMCaller.callVision({
-      schema: koreanHanjaMixSchema,
+      schema: koreanDocumentDetectionSchema,
       messages,
       primaryModel: model,
       fallbackModel: options?.fallbackModel,
@@ -6807,7 +6783,7 @@ var OcrStrategySampler = class {
       temperature: options?.temperature ?? 0,
       abortSignal: options?.abortSignal,
       component: "OcrStrategySampler",
-      phase: "korean-hanja-mix-detection"
+      phase: "korean-document-detection"
     });
     if (options?.aggregator) {
       options.aggregator.track(result.usage);
@@ -6815,10 +6791,9 @@ var OcrStrategySampler = class {
     const output = result.output;
     const normalizedLanguages = output.detectedLanguages.map(import_model.normalizeToBcp47).filter((tag) => tag !== null);
     this.logger.debug(
-      `[OcrStrategySampler] Page ${pageNo}: hasKoreanHanjaMix=${output.hasKoreanHanjaMix}, detectedLanguages=${normalizedLanguages.join(",")}`
+      `[OcrStrategySampler] Page ${pageNo}: detectedLanguages=${normalizedLanguages.join(",")}`
     );
     return {
-      hasKoreanHanjaMix: output.hasKoreanHanjaMix,
       detectedLanguages: normalizedLanguages
     };
   }
@@ -7067,16 +7042,7 @@ var VlmTextCorrector = class {
         pagesFailed: 0
       };
     }
-    let pageNumbers;
-    if (options?.koreanHanjaMixPages && options.koreanHanjaMixPages.length > 0) {
-      const hanjaSet = new Set(options.koreanHanjaMixPages);
-      pageNumbers = allPageNumbers.filter((p) => hanjaSet.has(p));
-      this.logger.info(
-        `[VlmTextCorrector] Filtering to ${pageNumbers.length} Korean-Hanja mix pages out of ${allPageNumbers.length} total`
-      );
-    } else {
-      pageNumbers = allPageNumbers;
-    }
+    const pageNumbers = allPageNumbers;
     const concurrency = options?.concurrency ?? DEFAULT_CONCURRENCY;
     this.logger.info(
       `[VlmTextCorrector] Processing ${pageNumbers.length} pages (concurrency: ${concurrency})...`
@@ -7281,7 +7247,7 @@ var VlmConversionPipeline = class {
    * Wrap the original callback with VLM text correction.
    * Returns a new callback that runs VLM correction before calling the original.
    */
-  wrapCallback(pdfPath, options, originalCallback, abortSignal, detectedLanguages, koreanHanjaMixPages) {
+  wrapCallback(pdfPath, options, originalCallback, abortSignal, detectedLanguages) {
     if (!options.vlmProcessorModel) {
       throw new Error("vlmProcessorModel is required when OCR strategy is VLM");
     }
@@ -7313,8 +7279,7 @@ var VlmConversionPipeline = class {
           abortSignal,
           onTokenUsage: options.onTokenUsage,
           documentLanguages: detectedLanguages,
-          pageTexts,
-          koreanHanjaMixPages
+          pageTexts
         }
       );
       await originalCallback(outputDir);
@@ -7466,8 +7431,7 @@ var PDFConverter = class {
         trackedOptions,
         postCorrectionCallback,
         abortSignal,
-        strategy.detectedLanguages,
-        strategy.koreanHanjaMixPages
+        strategy.detectedLanguages
       );
       const vlmOptions = strategy.detectedLanguages ? { ...trackedOptions, ocr_lang: strategy.detectedLanguages } : trackedOptions;
       await this.convert(
@@ -8002,7 +7966,7 @@ var META_DESCRIPTION_PATTERNS_EN = [
 var REPETITIVE_PATTERN_RATIO_THRESHOLD = 0.3;
 var REPETITIVE_PATTERN_MIN_REPEATS = 5;
 var HANGUL_REGEX2 = /[\uAC00-\uD7AF\u1100-\u11FF]/g;
-var CJK_REGEX2 = /[\u4E00-\u9FFF]/g;
+var CJK_REGEX = /[\u4E00-\u9FFF]/g;
 var VlmResponseValidator = class {
   /**
    * Validate VLM page result quality.
@@ -8070,7 +8034,7 @@ var VlmResponseValidator = class {
       return null;
     }
     const hangulCount = allContent.match(HANGUL_REGEX2)?.length ?? 0;
-    const cjkCount = allContent.match(CJK_REGEX2)?.length ?? 0;
+    const cjkCount = allContent.match(CJK_REGEX)?.length ?? 0;
     const koreanCjkCount = hangulCount + cjkCount;
     const ratio = koreanCjkCount / nonWhitespace.length;
     if (ratio < KOREAN_SCRIPT_RATIO_THRESHOLD) {