BabelDOC 0.5.16__tar.gz → 0.5.18__tar.gz

{babeldoc-0.5.16 → babeldoc-0.5.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: BabelDOC
-Version: 0.5.16
+Version: 0.5.18
 Summary: Yet Another Document Translator
 Project-URL: Homepage, https://github.com/funstory-ai/BabelDOC
 Project-URL: Issues, https://github.com/funstory-ai/BabelDOC/issues
@@ -90,7 +90,7 @@ Description-Content-Type: text/markdown
 
 PDF scientific paper translation and bilingual comparison library.
 
-- **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) 1000 free pages per month.
+- **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) Free usage quota is available; please refer to the FAQ section on the page for details.
 - **Self-deployment**: [PDFMathTranslate 2.0](https://github.com/PDFMathTranslate/PDFMathTranslate-next) support for BabelDOC, available for self-deployment + WebUI with more translation services.
 - Provides a simple [command line interface](#getting-started).
 - Provides a [Python API](#python-api).
@@ -268,6 +268,7 @@ uv run babeldoc --files example.pdf --files example2.pdf --openai --openai-model
 - `--openai-base-url`: Base URL for OpenAI API
 - `--openai-api-key`: API key for OpenAI service
 - `--enable-json-mode-if-requested`: Enable JSON mode for OpenAI requests (default: False)
+- `--term-pool-max-workers`: Maximum number of worker threads dedicated to automatic term extraction. If not specified, this defaults to the value of `--pool-max-workers`, which itself defaults to the QPS value when unset.
 
 > [!TIP]
 >
@@ -489,4 +490,4 @@ Everyone interacting in YADT and its sub-projects' codebases, issue trackers, ch
 > 2.  Then, during the scanned document detection phase (`DetectScannedFile` stage):
 >     *   If the document is identified as heavily scanned (e.g., >80% scanned pages) AND `auto_enable_ocr_workaround` is `true` (i.e., `translation_config.auto_enable_ocr_workaround` is true), the system will then attempt to set both `ocr_workaround` to `true` and `skip_scanned_detection` to `true`.
 >
-> This means that `--auto-enable-ocr-workaround` effectively gives the system control to enable OCR processing for scanned documents, potentially overriding manual settings for `--ocr-workaround` and `--skip_scanned_detection` based on its detection results. If the document is *not* detected as heavily scanned, then the initial `false` values for `ocr_workaround` and `skip_scanned_detection` (forced by `--auto-enable-ocr-workaround` at the `TranslationConfig` initialization stage) will remain in effect unless changed by other logic.
+> This means that `--auto-enable-ocr-workaround` effectively gives the system control to enable OCR processing for scanned documents, potentially overriding manual settings for `--ocr-workaround` and `--skip_scanned_detection` based on its detection results. If the document is *not* detected as heavily scanned, then the initial `false` values for `ocr_workaround` and `skip_scanned_detection` (forced by `--auto-enable-ocr-workaround` at the `TranslationConfig` initialization stage) will remain in effect unless changed by other logic.

{babeldoc-0.5.16 → babeldoc-0.5.18}/README.md

@@ -34,7 +34,7 @@
 
 PDF scientific paper translation and bilingual comparison library.
 
-- **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) 1000 free pages per month.
+- **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) Free usage quota is available; please refer to the FAQ section on the page for details.
 - **Self-deployment**: [PDFMathTranslate 2.0](https://github.com/PDFMathTranslate/PDFMathTranslate-next) support for BabelDOC, available for self-deployment + WebUI with more translation services.
 - Provides a simple [command line interface](#getting-started).
 - Provides a [Python API](#python-api).
@@ -212,6 +212,7 @@ uv run babeldoc --files example.pdf --files example2.pdf --openai --openai-model
 - `--openai-base-url`: Base URL for OpenAI API
 - `--openai-api-key`: API key for OpenAI service
 - `--enable-json-mode-if-requested`: Enable JSON mode for OpenAI requests (default: False)
+- `--term-pool-max-workers`: Maximum number of worker threads dedicated to automatic term extraction. If not specified, this defaults to the value of `--pool-max-workers`, which itself defaults to the QPS value when unset.
 
 > [!TIP]
 >
@@ -433,4 +434,4 @@ Everyone interacting in YADT and its sub-projects' codebases, issue trackers, ch
 > 2.  Then, during the scanned document detection phase (`DetectScannedFile` stage):
 >     *   If the document is identified as heavily scanned (e.g., >80% scanned pages) AND `auto_enable_ocr_workaround` is `true` (i.e., `translation_config.auto_enable_ocr_workaround` is true), the system will then attempt to set both `ocr_workaround` to `true` and `skip_scanned_detection` to `true`.
 >
-> This means that `--auto-enable-ocr-workaround` effectively gives the system control to enable OCR processing for scanned documents, potentially overriding manual settings for `--ocr-workaround` and `--skip_scanned_detection` based on its detection results. If the document is *not* detected as heavily scanned, then the initial `false` values for `ocr_workaround` and `skip_scanned_detection` (forced by `--auto-enable-ocr-workaround` at the `TranslationConfig` initialization stage) will remain in effect unless changed by other logic.
+> This means that `--auto-enable-ocr-workaround` effectively gives the system control to enable OCR processing for scanned documents, potentially overriding manual settings for `--ocr-workaround` and `--skip_scanned_detection` based on its detection results. If the document is *not* detected as heavily scanned, then the initial `false` values for `ocr_workaround` and `skip_scanned_detection` (forced by `--auto-enable-ocr-workaround` at the `TranslationConfig` initialization stage) will remain in effect unless changed by other logic.

babeldoc-0.5.18/babeldoc/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.5.18"

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/const.py

@@ -6,7 +6,7 @@ import subprocess
 import threading
 from pathlib import Path
 
-__version__ = "0.5.16"
+__version__ = "0.5.18"
 
 CACHE_FOLDER = Path.home() / ".cache" / "babeldoc"
 

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/format/pdf/babelpdf/cidfont.py

@@ -41,9 +41,12 @@ def get_descendant_fonts(doc, xref):
 
 
 def get_glyph_bbox(face, g):
-    face.load_glyph(g, freetype.FT_LOAD_NO_SCALE)
-    cbox = face.glyph.outline.get_bbox()
-    return cbox.xMin, cbox.yMin, cbox.xMax, cbox.yMax
+    try:
+        face.load_glyph(g, freetype.FT_LOAD_NO_SCALE)
+        cbox = face.glyph.outline.get_bbox()
+        return cbox.xMin, cbox.yMin, cbox.xMax, cbox.yMax
+    except Exception:
+        return 0, 0, 0, 0
 
 
 def get_face_bbox(blob):

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/format/pdf/document_il/frontend/il_creater.py

@@ -3,12 +3,14 @@ import functools
 import logging
 import math
 import re
+import unicodedata
 from io import BytesIO
 from itertools import islice
 from typing import Literal
 
 import freetype
 import pymupdf
+import tiktoken
 
 import babeldoc.pdfminer.pdfinterp
 from babeldoc.format.pdf.babelpdf.base14 import get_base14_bbox
@@ -18,6 +20,7 @@ from babeldoc.format.pdf.babelpdf.encoding import get_type1_encoding
 from babeldoc.format.pdf.babelpdf.utils import guarded_bbox
 from babeldoc.format.pdf.document_il import il_version_1
 from babeldoc.format.pdf.document_il.utils import zstd_helper
+from babeldoc.format.pdf.document_il.utils.fontmap import FontMapper
 from babeldoc.format.pdf.document_il.utils.matrix_helper import decompose_ctm
 from babeldoc.format.pdf.document_il.utils.style_helper import BLACK
 from babeldoc.format.pdf.document_il.utils.style_helper import YELLOW
@@ -358,6 +361,10 @@ class ILCreater:
         self.render_order = 0
         self.current_clip_paths: list[tuple] = []
         self.clip_paths_stack: list[list[tuple]] = []
+        # For valid character collection
+        self.font_mapper = FontMapper(translation_config)
+        self.tokenizer = tiktoken.encoding_for_model("gpt-4o")
+        self._page_valid_chars_buffer: list[str] | None = None
 
     def transform_clip_path(
         self,
@@ -566,8 +573,32 @@ class ILCreater:
         self.current_clip_paths = []
         self.clip_paths_stack = []
         self.docs.page.append(self.current_page)
+        # Prepare per-page buffer for valid characters on translated pages
+        self._page_valid_chars_buffer = []
 
     def on_page_end(self):
+        # Accumulate this page's valid characters and tokens into shared context
+        try:
+            if (
+                self._page_valid_chars_buffer is not None
+                and len(self._page_valid_chars_buffer) > 0
+            ):
+                page_text = "".join(self._page_valid_chars_buffer)
+                char_count = len(page_text)
+                try:
+                    token_count = len(
+                        self.tokenizer.encode(page_text, disallowed_special=())
+                    )
+                except Exception as e:
+                    logger.warning("Failed to compute token count for page: %s", e)
+                    token_count = 0
+                self.translation_config.shared_context_cross_split_part.add_valid_counts(
+                    char_count, token_count
+                )
+        except Exception as e:
+            logger.warning("Failed to accumulate page valid stats: %s", e)
+        finally:
+            self._page_valid_chars_buffer = []
         self.progress.advance(1)
 
     def on_page_crop_box(
@@ -848,6 +879,11 @@ class ILCreater:
                 "Failed to get rotation angle for char %s",
                 char.get_text(),
             )
+        # Collect valid characters for statistics
+        try:
+            self._collect_valid_char(char.get_text())
+        except Exception as e:
+            logger.warning("Error collecting valid char: %s", e)
         gs = self.create_graphic_state(char.graphicstate)
         # Get font from current page or xobject
         font = None
@@ -983,6 +1019,43 @@ class ILCreater:
                 )
             )
 
+    def _collect_valid_char(self, ch: str):
+        """Append a valid character into the current page buffer according to rules.
+        Rules:
+        - Include whitespace matched by space_regex directly.
+        - Ignore categories that are never normal text: {Cc, Cs, Co, Cn}.
+        - Apply inverted criteria from formular_helper.py (21-28):
+          empty -> invalid, contains '(cid:' -> invalid,
+          not has_char(ch) -> invalid unless len(ch) > 1 and all(has_char(x)).
+        """
+        if self._page_valid_chars_buffer is None:
+            return
+        if space_regex.match(ch):
+            self._page_valid_chars_buffer.append(ch)
+            return
+        try:
+            cat = unicodedata.category(ch[0]) if ch else None
+        except Exception:
+            cat = None
+        if cat in {"Cc", "Cs", "Co", "Cn"}:
+            return
+        is_invalid = False
+        if not ch:
+            is_invalid = True
+        elif "(cid:" in ch:
+            is_invalid = True
+        else:
+            try:
+                if not self.font_mapper.has_char(ch):
+                    if len(ch) > 1 and all(self.font_mapper.has_char(x) for x in ch):
+                        is_invalid = False
+                    else:
+                        is_invalid = True
+            except Exception:
+                is_invalid = True
+        if not is_invalid:
+            self._page_valid_chars_buffer.append(ch)
+
     def on_lt_curve(self, curve: babeldoc.pdfminer.layout.LTCurve):
         if not self.enable_graphic_element_process:
             return

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/format/pdf/document_il/midend/automatic_term_extractor.py

@@ -342,8 +342,12 @@ class AutomaticTermExtractor:
             self.stage_name,
             total,
         ) as pbar:
+            max_workers = self.translation_config.term_pool_max_workers
+            logger.info(
+                f"Using {max_workers} worker threads for automatic term extraction."
+            )
             with PriorityThreadPoolExecutor(
-                max_workers=self.translation_config.pool_max_workers,
+                max_workers=max_workers,
             ) as executor:
                 for page in doc_il.page:
                     self.process_page(page, executor, pbar, tracker.new_page())

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/format/pdf/high_level.py

@@ -755,6 +755,23 @@ def do_translate(
         logger.info(
             f"finish translate: {original_pdf_path}, cost: {finish_time - start_time} s",
         )
+        # Populate aggregate valid text statistics into result
+        try:
+            sc = translation_config.shared_context_cross_split_part
+            result.total_valid_character_count = getattr(
+                sc, "valid_char_count_total", 0
+            )
+            token_total = getattr(sc, "total_valid_text_token_count", None)
+            result.total_valid_text_token_count = (
+                token_total if isinstance(token_total, int) else 0
+            )
+        except Exception as e:
+            logger.warning("Failed to populate valid text statistics: %s", e)
+            try:
+                result.total_valid_character_count = 0
+                result.total_valid_text_token_count = 0
+            except Exception:
+                pass
         result.original_pdf_path = translation_config.input_file
         result.peak_memory_usage = peak_memory_usage
 

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/format/pdf/translation_config.py

@@ -32,6 +32,9 @@ class SharedContextCrossSplitPart:
         self.auto_extracted_glossary: Glossary | None = None
         self.raw_extracted_terms: list[tuple[str, str]] = []
         self.auto_enabled_ocr_workaround = False
+        # Statistics for valid characters/text across the whole file
+        self.valid_char_count_total: int = 0
+        self.total_valid_text_token_count: int = 0
 
     def initialize_glossaries(self, initial_glossaries: list[Glossary] | None):
         with self._lock:
@@ -45,6 +48,9 @@ class SharedContextCrossSplitPart:
             for g in self.user_glossaries:
                 for entity in g.normalized_lookup:
                     self.norm_terms.add(entity)
+            # reset statistics buffer when initializing
+            self.valid_char_count_total = 0
+            self.total_valid_text_token_count = 0
 
     def add_raw_extracted_term_pair(self, src: str, tgt: str):
         with self._lock:
@@ -67,7 +73,11 @@ class SharedContextCrossSplitPart:
         return current_name
 
     def contains_term(self, term: str) -> bool:
-        pass
+        with self._lock:
+            try:
+                return term in self.norm_terms
+            except Exception:
+                return False
 
     def finalize_auto_extracted_glossary(self):
         with self._lock:
@@ -112,12 +122,24 @@ class SharedContextCrossSplitPart:
                     all_glossaries.append(self.auto_extracted_glossary)
                 return all_glossaries
 
+    def add_valid_counts(self, char_count: int, token_count: int):
+        """Accumulate valid character and token counts in a threadsafe way."""
+        if char_count <= 0 and token_count <= 0:
+            return
+        with self._lock:
+            if char_count > 0:
+                self.valid_char_count_total += char_count
+            if token_count > 0:
+                self.total_valid_text_token_count += token_count
+
 
 class TranslationConfig:
     @staticmethod
     def create_max_pages_per_part_split_strategy(max_pages_per_part: int):
         return PageCountStrategy(max_pages_per_part)
 
+    # for backward compatibility,
+    # new parameters should be added at the end of the function.
     def __init__(
         self,
         translator: BaseTranslator,
@@ -176,6 +198,7 @@ class TranslationConfig:
         skip_formula_offset_calculation: bool = False,
         term_extraction_translator: BaseTranslator | None = None,
         metadata_extra_data: str | None = None,
+        term_pool_max_workers: int | None = None,
     ):
         self.translator = translator
         self.term_extraction_translator = term_extraction_translator or translator
@@ -204,6 +227,13 @@ class TranslationConfig:
         self.pool_max_workers = (
             pool_max_workers if pool_max_workers is not None else qps
         )
+        # Set term_pool_max_workers for automatic term extraction.
+        # If not provided, default to pool_max_workers.
+        self.term_pool_max_workers = (
+            term_pool_max_workers
+            if term_pool_max_workers is not None
+            else self.pool_max_workers
+        )
         self.split_short_lines = split_short_lines
 
         self.short_line_split_factor = short_line_split_factor
@@ -463,6 +493,8 @@ class TranslateResult:
     no_watermark_dual_pdf_path: Path | None
     peak_memory_usage: int | None
     auto_extracted_glossary_path: Path | None
+    total_valid_character_count: int | None
+    total_valid_text_token_count: int | None
 
     def __init__(
         self,
@@ -479,6 +511,8 @@ class TranslateResult:
         self.no_watermark_dual_pdf_path = dual_pdf_path
 
         self.auto_extracted_glossary_path = auto_extracted_glossary_path
+        self.total_valid_character_count = None
+        self.total_valid_text_token_count = None
 
     def __str__(self):
         """Return a human-readable string representation of the translation result."""
@@ -524,6 +558,20 @@ class TranslateResult:
         if hasattr(self, "peak_memory_usage") and self.peak_memory_usage:
             result.append(f"\tPeak memory usage: {self.peak_memory_usage} MB")
 
+        if hasattr(self, "total_valid_character_count") and isinstance(
+            self.total_valid_character_count, int
+        ):
+            result.append(
+                f"\tTotal valid character count: {self.total_valid_character_count}"
+            )
+
+        if hasattr(self, "total_valid_text_token_count") and isinstance(
+            self.total_valid_text_token_count, int
+        ):
+            result.append(
+                f"\tTotal valid text token count (gpt-4o): {self.total_valid_text_token_count}"
+            )
+
         if result:
             result.insert(0, "Translation results:")
 

{babeldoc-0.5.16 → babeldoc-0.5.18}/babeldoc/main.py

@@ -26,7 +26,7 @@ from babeldoc.translator.translator import OpenAITranslator
 from babeldoc.translator.translator import set_translate_rate_limiter
 
 logger = logging.getLogger(__name__)
-__version__ = "0.5.16"
+__version__ = "0.5.18"
 
 
 def create_parser():
@@ -280,6 +280,11 @@ def create_parser():
         type=int,
         help="Maximum number of worker threads for internal task processing pools. If not specified, defaults to QPS value. This parameter directly sets the worker count, replacing previous QPS-based dynamic calculations.",
     )
+    translation_group.add_argument(
+        "--term-pool-max-workers",
+        type=int,
+        help="Maximum number of worker threads dedicated to automatic term extraction. If not specified, defaults to --pool-max-workers (or QPS value when unset).",
+    )
     translation_group.add_argument(
         "--no-auto-extract-glossary",
         action="store_false",
@@ -694,6 +699,7 @@ async def main():
             figure_table_protection_threshold=args.figure_table_protection_threshold,
             skip_formula_offset_calculation=args.skip_formula_offset_calculation,
             metadata_extra_data=args.metadata_extra_data,
+            term_pool_max_workers=args.term_pool_max_workers,
         )
 
         def nop(_x):

{babeldoc-0.5.16 → babeldoc-0.5.18}/docs/CONTRIBUTOR_REWARD.md

@@ -1,4 +1,4 @@
-# BabelDOC/PDFMathTranslate 贡献者奖励规则
+# BabelDOC/PDFMathTranslate/OneAIFW 贡献者奖励规则
 
 ## 月度活跃贡献者奖励规则
 
@@ -21,6 +21,7 @@
      2. [Byaidu/PDFMathTranslate](https://github.com/Byaidu/PDFMathTranslate/pulls) 的主分支 
      3. [PDFMathTranslate/PDFMathTranslate-next](https://github.com/PDFMathTranslate/PDFMathTranslate-next/pulls) 的主分支。
      4. [guaguastandup/zotero-pdf2zh](https://github.com/guaguastandup/zotero-pdf2zh) 的主分支
+     5. [funstory-ai/aifw](https://github.com/funstory-ai/aifw) 的主分支
    - 若目标为 [funstory-ai/BabelDOC](https://github.com/funstory-ai/BabelDOC/pulls) 的 PR 未被合并，但被维护者认定为有价值的概念验证，同样符合条件。
    - 审核 PR、撰写 wiki 等贡献也必须是以上两个仓库。
    - 同一贡献者每月仅可申请一次（无论提交 PR 数量）

{babeldoc-0.5.16 → babeldoc-0.5.18}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "BabelDOC"
-version = "0.5.16"
+version = "0.5.18"
 description = "Yet Another Document Translator"
 license = "AGPL-3.0"
 readme = "README.md"
@@ -162,7 +162,7 @@ pythonpath = [".", "src"]
 testpaths = ["tests"]
 
 [bumpver]
-current_version = "0.5.16"
+current_version = "0.5.18"
 version_pattern = "MAJOR.MINOR.PATCH[.PYTAGNUM]"
 
 [bumpver.file_patterns]

babeldoc-0.5.16/babeldoc/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.5.16"