pdf-kintsugi 0.1.0__tar.gz

pdf_kintsugi-0.1.0/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.3
+Name: pdf-kintsugi
+Version: 0.1.0
+Summary: PDF table extraction with Docling and pdfplumber
+Author: moss (moss-tms)
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Dist: docling>=2.73.0
+Requires-Dist: pdfplumber>=0.11.9
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# pdf-kintsugi
+
+**pdf-kintsugi** is a Python library that PDF table extraction and text recovery by combining the high-level document understanding of **Docling** with the precise, character-level bounding box analysis of **pdfplumber**.
+
+Like the Japanese art of *kintsugi* (repairing broken pottery with gold), this library takes the initial parsing results from Docling and "repairs" complex table structures and text with garbled characters using pdfplumber's precise geometric layout analysis.
+
+## Features
+
+- **Table Parsing:** Improves Docling's table extraction by using pdfplumber to detect omitted geometric lines and infer table edges.
+- **Text Correction:** Optionally corrects garbled text using fine-grained character bounding boxes.
+- **Seamless Integration with Docling:** Acts as a post-processor for Docling. It takes a `ConversionResult` from Docling, enhances the tables and text in place, and returns the updated document model.
+
+## Installation
+
+The project requires **Python 3.10+**.
+
+You can install it using `pip` or your favorite package manager:
+
+```bash
+pip install pdf-kintsugi
+```
+
+## Quick Start
+
+`pdf-kintsugi` works alongside `docling`. Here is a basic example of how to use `PDFKintsugi`:
+
+```python
+from docling.document_converter import DocumentConverter
+from pdf_kintsugi import PDFKintsugi
+
+pdf_path = "path/to/your/document.pdf"
+
+# 1. Parse the document using Docling first
+converter = DocumentConverter()
+docling_result = converter.convert(pdf_path)
+
+# 2. Initialize PDFKintsugi with the source PDF and the Docling result
+kintsugi_parser = PDFKintsugi(
+    source=pdf_path,
+    docling_result=docling_result,
+    tolerance=3.0,          # Adjust merging tolerance for table lines
+    replace_text=False,     # Set to True to enable text/character correction
+    replace_table=True      # Set to True to enable table structure repair
+)
+
+# 3. Get the parsing result
+kintsugi_result = kintsugi_parser.parse()
+
+# Now you can use kintsugi_result just like a regular Docling document
+print(kintsugi_result.document.export_to_markdown())
+```
+
+## Configuration
+
+The `PDFKintsugi` class accepts several parameters to tune the extraction:
+
+- `source` (`str`): The file path to the source PDF document.
+- `docling_result` (`ConversionResult`): The parsed document object returned by Docling.
+- `tolerance` (`float`, default `3.0`): The line-merging tolerance. A smaller value (e.g., `1.5`) helps prevent adjacent tables from being incorrectly merged, while a larger value (e.g., `5.0`) can help stitch together fragmented tables.
+- `replace_text` (`bool`, default `False`): If `True`, utilizes `pdfplumber` to extract characters and correct garbled text.
+- `replace_table` (`bool`, default `True`): If `True`, rebuilds and overrides the Docling table representations using geometric line intersections and edge inference.
+
+## Contributing
+
+Contributions are welcome! This project uses `uv` for dependency management.
+
+1. Clone the repository.
+2. Setup the environment: `uv sync`
+3. Run tests before submitting a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

pdf_kintsugi-0.1.0/README.md

@@ -0,0 +1,73 @@
+# pdf-kintsugi
+
+**pdf-kintsugi** is a Python library that PDF table extraction and text recovery by combining the high-level document understanding of **Docling** with the precise, character-level bounding box analysis of **pdfplumber**.
+
+Like the Japanese art of *kintsugi* (repairing broken pottery with gold), this library takes the initial parsing results from Docling and "repairs" complex table structures and text with garbled characters using pdfplumber's precise geometric layout analysis.
+
+## Features
+
+- **Table Parsing:** Improves Docling's table extraction by using pdfplumber to detect omitted geometric lines and infer table edges.
+- **Text Correction:** Optionally corrects garbled text using fine-grained character bounding boxes.
+- **Seamless Integration with Docling:** Acts as a post-processor for Docling. It takes a `ConversionResult` from Docling, enhances the tables and text in place, and returns the updated document model.
+
+## Installation
+
+The project requires **Python 3.10+**.
+
+You can install it using `pip` or your favorite package manager:
+
+```bash
+pip install pdf-kintsugi
+```
+
+## Quick Start
+
+`pdf-kintsugi` works alongside `docling`. Here is a basic example of how to use `PDFKintsugi`:
+
+```python
+from docling.document_converter import DocumentConverter
+from pdf_kintsugi import PDFKintsugi
+
+pdf_path = "path/to/your/document.pdf"
+
+# 1. Parse the document using Docling first
+converter = DocumentConverter()
+docling_result = converter.convert(pdf_path)
+
+# 2. Initialize PDFKintsugi with the source PDF and the Docling result
+kintsugi_parser = PDFKintsugi(
+    source=pdf_path,
+    docling_result=docling_result,
+    tolerance=3.0,          # Adjust merging tolerance for table lines
+    replace_text=False,     # Set to True to enable text/character correction
+    replace_table=True      # Set to True to enable table structure repair
+)
+
+# 3. Get the parsing result
+kintsugi_result = kintsugi_parser.parse()
+
+# Now you can use kintsugi_result just like a regular Docling document
+print(kintsugi_result.document.export_to_markdown())
+```
+
+## Configuration
+
+The `PDFKintsugi` class accepts several parameters to tune the extraction:
+
+- `source` (`str`): The file path to the source PDF document.
+- `docling_result` (`ConversionResult`): The parsed document object returned by Docling.
+- `tolerance` (`float`, default `3.0`): The line-merging tolerance. A smaller value (e.g., `1.5`) helps prevent adjacent tables from being incorrectly merged, while a larger value (e.g., `5.0`) can help stitch together fragmented tables.
+- `replace_text` (`bool`, default `False`): If `True`, utilizes `pdfplumber` to extract characters and correct garbled text.
+- `replace_table` (`bool`, default `True`): If `True`, rebuilds and overrides the Docling table representations using geometric line intersections and edge inference.
+
+## Contributing
+
+Contributions are welcome! This project uses `uv` for dependency management.
+
+1. Clone the repository.
+2. Setup the environment: `uv sync`
+3. Run tests before submitting a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

pdf_kintsugi-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[project]
+name = "pdf-kintsugi"
+version = "0.1.0"
+description = "PDF table extraction with Docling and pdfplumber"
+readme = "README.md"
+license = { text = "MIT" }
+authors = [
+    { name = "moss (moss-tms)" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+]
+requires-python = ">=3.10"
+dependencies = [
+    "docling>=2.73.0",
+    "pdfplumber>=0.11.9",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.9,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 88
+target-version = "py310"
+
+[tool.ruff.lint]
+select = [
+    "E", "F",
+    "I",
+]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+[dependency-groups]
+dev = [
+    "ruff>=0.15.8",
+]

pdf_kintsugi-0.1.0/src/pdf_kintsugi/__init__.py

@@ -0,0 +1,6 @@
+"""pdf-kintsugi - Enhanced PDF table extraction with Docling and pdfplumber."""
+
+from pdf_kintsugi.parser import PDFKintsugi
+
+__version__ = "0.1.0"
+__all__ = ["PDFKintsugi"]

pdf_kintsugi-0.1.0/src/pdf_kintsugi/line/extractor.py

@@ -0,0 +1,203 @@
+from operator import itemgetter
+
+from pdfplumber.page import Page
+
+
+class LineExtractor:
+    def __init__(self, page: Page | None, lines: list[dict] | None = None) -> None:
+        self.lines: list[dict] = []
+        self.horizontal_lines: list[dict] = []
+        self.vertical_lines: list[dict] = []
+
+        if page:
+            self.init_from_page(page)
+        if lines:
+            self.init_from_lines(lines)
+
+    def init_from_page(self, page: Page) -> None:
+        self.extract_lines(page.lines + page.rects + page.curves + page.images)
+        self.extract_rects(page.rects)
+
+    def init_from_lines(self, lines: list[dict]) -> None:
+        self.extract_lines(lines)
+
+    def get_orientation(self, line: dict) -> str:
+        width = abs(line["x1"] - line["x0"])
+        height = abs(line["bottom"] - line["top"])
+
+        ratio = 1.5
+        max_thickness = 5.0
+
+        if width > height * ratio and height < max_thickness:
+            return "x"
+        if height > width * ratio and width < max_thickness:
+            return "y"
+
+        return "none"
+
+    def extract_lines(self, lines: list[dict]) -> None:
+        """we process page.lines, page.rects, page.curves, page.images in bulk"""
+        for line in lines:
+            orientation = self.get_orientation(line)
+            if orientation == "x":
+                rep_y = (line["top"] + line["bottom"]) / 2
+                self.horizontal_lines.append(
+                    {
+                        "x0": line["x0"],
+                        "x1": line["x1"],
+                        "top": rep_y,
+                        "bottom": rep_y,
+                        "orientation": orientation,
+                    }
+                )
+            if orientation == "y":
+                rep_x = (line["x0"] + line["x1"]) / 2
+                self.vertical_lines.append(
+                    {
+                        "x0": rep_x,
+                        "x1": rep_x,
+                        "top": line["top"],
+                        "bottom": line["bottom"],
+                        "orientation": orientation,
+                    }
+                )
+
+    def extract_rects(self, rects: list[dict]) -> None:
+        """we extract page.rects which can be disassembled to 4 lines"""
+        for rect in rects:
+            if self.get_orientation(rect) != "none" or (
+                rect["width"] < 5.0 and rect["height"] < 5.0
+            ):
+                continue
+
+            # 枠線がない or 色付けのためのrectならばcontinue
+            if rect["stroking_color"] is None or rect["fill"] is True:
+                continue
+
+            for y_key in ["top", "bottom"]:
+                self.horizontal_lines.append(
+                    {
+                        "x0": rect["x0"],
+                        "x1": rect["x1"],
+                        "top": rect[y_key],
+                        "bottom": rect[y_key],
+                        "orientation": "x",
+                    }
+                )
+
+            for x_key in ["x0", "x1"]:
+                self.vertical_lines.append(
+                    {
+                        "x0": rect[x_key],
+                        "x1": rect[x_key],
+                        "top": rect["top"],
+                        "bottom": rect["bottom"],
+                        "orientation": "y",
+                    }
+                )
+
+    def can_merge(
+        self, line1_: dict, line2_: dict, main_coord, sub_start, sub_end, tolerance=1.5
+    ) -> bool:
+        """whether two lines can be merged"""
+        if line1_[sub_start] < line2_[sub_start]:
+            line1, line2 = line1_, line2_
+        else:
+            line1, line2 = line2_, line1_
+
+        is_collinear = abs(line1[main_coord] - line2[main_coord]) <= tolerance
+        is_connected = line2[sub_start] <= line1[sub_end] + tolerance
+
+        return is_collinear and is_connected
+
+    def merge_lines(
+        self, lines: list[dict], orientation: str, tolerance=1.5
+    ) -> list[dict]:
+        """merge collinear line segments"""
+        if not lines:
+            return []
+
+        # 1. lineを座標でsort
+        if orientation == "y":
+            # 1. y座標方向のlineをmergeしたい。x座標が等しいものを寄せる、y座標でsrot
+            lines.sort(key=itemgetter("x0", "top"))
+            main_coord = "x0"
+            sub_start = "top"
+            sub_end = "bottom"
+        else:
+            # 2. x座標方向のlineをmergeしたい。y座標が等しいものを寄せる、x座標でsrot
+            lines.sort(key=itemgetter("top", "x0"))
+            main_coord = "top"
+            sub_start = "x0"
+            sub_end = "x1"
+
+        merged = []
+        current = lines[0]
+
+        for next_line in lines[1:]:
+            if self.can_merge(
+                current, next_line, main_coord, sub_start, sub_end, tolerance
+            ):
+                current[sub_start] = min(current[sub_start], next_line[sub_start])
+                current[sub_end] = max(current[sub_end], next_line[sub_end])
+            else:
+                merged.append(current)
+                current = next_line
+
+        merged.append(current)
+        return merged
+
+    def remove_included_lines(
+        self, lines: list[dict], orientation: str, tolerance=1.5
+    ) -> list[dict]:
+        if orientation == "y":
+            # 1. y座標方向のlineをmergeしたい。x座標が等しいものを寄せる
+            main_coord = "x0"
+            sub_start = "top"
+            sub_end = "bottom"
+        else:
+            # 2. x座標方向のlineをmergeしたい。y座標が等しいものを寄せる
+            main_coord = "top"
+            sub_start = "x0"
+            sub_end = "x1"
+
+        skip = set()
+        for i, long in enumerate(lines):
+            for j, short in enumerate(lines):
+                if i in skip or j in skip or i == j:
+                    continue
+
+                is_collinear = abs(long[main_coord] - short[main_coord]) <= tolerance
+                include_start = long[sub_start] - tolerance <= short[sub_start]
+                include_end = short[sub_end] <= long[sub_end] + tolerance
+
+                if is_collinear and include_start and include_end:
+                    skip.add(j)
+
+        extracted = [line for i, line in enumerate(lines) if i not in skip]
+        return extracted
+
+    def extract_by_length(self, min_line_length=3.0) -> list[dict]:
+        """exclude the lines whose length is shorter than min_line_lenght"""
+        extract_lines: list[dict] = []
+        for line in self.lines:
+            orientation = line["orientation"]
+            length = (
+                line["x1"] - line["x0"]
+                if orientation == "x"
+                else line["bottom"] - line["top"]
+            )
+            if length >= min_line_length:
+                extract_lines.append(line)
+
+        return extract_lines
+
+    def extract(self) -> list[dict]:
+        """extract the lines contained in the page"""
+        merged_horizontal = self.merge_lines(self.horizontal_lines, "x")
+        merged_vertical = self.merge_lines(self.vertical_lines, "y")
+
+        self.lines = merged_horizontal + merged_vertical
+        self.lines = self.extract_by_length()
+
+        return self.lines

pdf_kintsugi-0.1.0/src/pdf_kintsugi/line/merger.py

@@ -0,0 +1,136 @@
+from pdf_kintsugi.utils.coord import is_same
+from pdf_kintsugi.utils.union_find import UnionFind
+
+
+class LineMerger:
+    def __init__(self, lines: list[dict], tolerance: float) -> None:
+        self.lines: list[dict] = lines
+        self.tolerance: float = tolerance
+
+    def merge(self) -> list[list[dict]]:
+        line_num = len(self.lines)
+        uf = UnionFind(line_num)
+
+        for i in range(line_num):
+            for j in range(i + 1, line_num):
+                if self._is_crossing_lines(i, j):
+                    uf.merge(i, j)
+
+        self.group_ids: list[list[int]] = uf.groups()
+        self.groups: list[list[dict]] = [
+            [self.lines[id] for id in group] for group in self.group_ids
+        ]
+
+        self._infer_table_edges()
+
+        return self.groups
+
+    def _infer_table_edges(self) -> None:
+        core_groups = []
+        loose_lines = []
+
+        for group in self.groups:
+            has_x = any(line["orientation"] == "x" for line in group)
+            has_y = any(line["orientation"] == "y" for line in group)
+            if has_x and has_y and len(group) > 4:
+                core_groups.append(group)
+            elif len(group) == 1:
+                loose_lines.extend(group)
+
+        for group in core_groups:
+            x_lines = [line for line in group if line["orientation"] == "x"]
+            y_lines = [line for line in group if line["orientation"] == "y"]
+
+            logical_left = min(line["x0"] for line in x_lines)
+            logical_right = max(line["x1"] for line in x_lines)
+            logical_top = min(line["top"] for line in y_lines)
+            logical_bottom = max(line["bottom"] for line in y_lines)
+
+            # Check y-edges (left and right)
+            for target_x in (logical_left, logical_right):
+                has_edge = any(is_same(line["x0"], target_x, 1.5) for line in y_lines)
+
+                if not has_edge:
+                    found = False
+                    for i, loose in enumerate(loose_lines):
+                        if loose["orientation"] == "y" and is_same(
+                            loose["x0"], target_x, self.tolerance
+                        ):
+                            overlap = min(loose["bottom"], logical_bottom) - max(
+                                loose["top"], logical_top
+                            )
+                            if overlap > 0:
+                                group.append(loose)
+                                y_lines.append(loose)
+                                loose_lines.pop(i)
+                                found = True
+                                break
+                    if not found:
+                        interpolated = {
+                            "orientation": "y",
+                            "x0": target_x,
+                            "x1": target_x,
+                            "top": logical_top,
+                            "bottom": logical_bottom,
+                        }
+                        group.append(interpolated)
+                        y_lines.append(interpolated)
+
+            # Check x-edges (top and bottom)
+            for target_y in (logical_top, logical_bottom):
+                has_edge = any(is_same(line["top"], target_y, 1.5) for line in x_lines)
+                if not has_edge:
+                    found = False
+                    for i, loose in enumerate(loose_lines):
+                        if loose["orientation"] == "x" and is_same(
+                            loose["top"], target_y, self.tolerance
+                        ):
+                            overlap = min(loose["x1"], logical_right) - max(
+                                loose["x0"], logical_left
+                            )
+                            if overlap > 0:
+                                group.append(loose)
+                                x_lines.append(loose)
+                                loose_lines.pop(i)
+                                found = True
+                                break
+                    if not found:
+                        interpolated = {
+                            "orientation": "x",
+                            "x0": logical_left,
+                            "x1": logical_right,
+                            "top": target_y,
+                            "bottom": target_y,
+                        }
+                        group.append(interpolated)
+                        x_lines.append(interpolated)
+
+        self.groups = core_groups
+        if loose_lines:
+            self.groups.extend([[line] for line in loose_lines])
+
+    def _is_crossing_lines(self, id1: int, id2: int, tolerance: float = 1.5) -> bool:
+        if self.lines[id1]["orientation"] == self.lines[id2]["orientation"]:
+            return False
+
+        if self.lines[id1]["orientation"] == "y":
+            id1, id2 = id2, id1
+
+        x_line = self.lines[id1]
+        y_line = self.lines[id2]
+
+        return (
+            x_line["x0"] - tolerance <= y_line["x0"] <= x_line["x1"] + tolerance
+            and y_line["top"] - tolerance
+            <= x_line["top"]
+            <= y_line["bottom"] + tolerance
+        )
+
+    def extract_independent_horizontal_lines(self) -> list[dict]:
+        independent_horizontal_lines = [
+            self.lines[group[0]]
+            for group in self.group_ids
+            if len(group) == 1 and self.lines[group[0]]["orientation"] == "x"
+        ]
+
+        return independent_horizontal_lines

pdf_kintsugi-0.1.0/src/pdf_kintsugi/page_parser.py

@@ -0,0 +1,104 @@
+from pdfplumber.page import Page
+
+from pdf_kintsugi.line.extractor import LineExtractor
+from pdf_kintsugi.line.merger import LineMerger
+from pdf_kintsugi.table.detector import TableDetector
+from pdf_kintsugi.table.merger import TableMerger
+from pdf_kintsugi.table.table import Table
+from pdf_kintsugi.text.char_extractor import CharExtractor
+
+
+class PageParser:
+    def __init__(self, page: Page, docling_tables: list[dict], tolerance) -> None:
+        self.page: Page = page
+        self.page_number: int = page.page_number
+        self.doclings: list[dict] = docling_tables
+        self.tol: float = tolerance
+
+        char_extractor = CharExtractor(page)
+        self.chars: list[dict] = char_extractor.extract()
+
+        line_extractor = LineExtractor(self.page)
+        self.lines: list[dict] = line_extractor.extract()
+
+        self.build(counter=0)
+
+    def build(self, counter: int = 1) -> list[list[Table]]:
+        if counter == 1:
+            line_extractor = LineExtractor(None, self.lines)
+            self.lines = line_extractor.extract()
+
+        my_tables: list[Table] = self._detect_tables()
+        links: list[list[int]] = self._link_docling(my_tables)
+
+        # doclingとtableが1対1対応しているやつを解析する, それ以外のmy tableは不要
+        valid_indices = [i for i in range(len(my_tables)) if len(links[i]) == 1]
+        my_tables = [my_tables[i] for i in valid_indices]
+        links = [links[i] for i in valid_indices]
+        build_success_tables: list[Table] = []
+
+        # build
+        for i, table in enumerate(my_tables):
+            docling = self.doclings[links[i][0]]
+            success = table.build(self.chars, docling)
+
+            if success:
+                build_success_tables.append(table)
+
+        my_tables = build_success_tables
+        links = self._link_docling(my_tables)
+
+        if counter == 0:
+            self._merge_tables(my_tables, links)
+            return []
+
+        rev_links: list[list[Table]] = [[] for _ in range(len(self.doclings))]
+        for my_idx, doc_idx in enumerate(links):
+            rev_links[doc_idx[0]].append(my_tables[my_idx])
+
+        return rev_links
+
+    def _merge_tables(self, my_tables: list[Table], links: list[list[int]]) -> None:
+        same_cluster: dict[int, list[Table]] = {}
+        for i, table in enumerate(my_tables):
+            idx = links[i][0]
+            if idx not in same_cluster:
+                same_cluster[idx] = []
+            same_cluster[idx].append(table)
+
+        table_merger = TableMerger()
+        for tables in same_cluster.values():
+            sz = len(tables)
+            for i in range(sz):
+                for j in range(i + 1, sz):
+                    table_merger.merge(tables[i], tables[j])
+
+    def _link_docling(self, tables: list[Table]) -> list[list[int]]:
+        """
+        docling_table内にあるmy_tableを紐づける, my_tableがkeyでdoclingのindexがvalue
+        """
+        links: list[list[int]] = [
+            [
+                i
+                for i, doc_table in enumerate(self.doclings)
+                if max(doc_table["bbox"][0], table.bbox[0])
+                < min(doc_table["bbox"][2], table.bbox[2])
+                and max(doc_table["bbox"][1], table.bbox[1])
+                < min(doc_table["bbox"][3], table.bbox[3])
+            ]
+            for table in tables
+        ]
+
+        return links
+
+    def _detect_tables(self) -> list[Table]:
+        line_groups = self._merge_lines()
+
+        table_detector = TableDetector(line_groups, self.page_number, self.tol)
+        tables: list[Table] = table_detector.detect()
+        return tables
+
+    def _merge_lines(self) -> list[list[dict]]:
+        line_merger = LineMerger(self.lines, self.tol)
+        line_groups: list[list[dict]] = line_merger.merge()
+        return line_groups