docling 0.1.2__tar.gz → 1.1.2__tar.gz

docling-1.1.2/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.1
+Name: docling
+Version: 1.1.2
+Summary: Docling PDF conversion package
+Home-page: https://github.com/DS4SD/docling
+License: MIT
+Keywords: docling,convert,document,pdf,layout model,segmentation,table structure,table former
+Author: Christoph Auer
+Author-email: cau@zurich.ibm.com
+Requires-Python: >=3.10,<4.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Provides-Extra: easyocr
+Provides-Extra: ocr
+Requires-Dist: deepsearch-glm (>=0.19.0,<1)
+Requires-Dist: docling-core (>=1.1.2,<2.0.0)
+Requires-Dist: docling-ibm-models (>=1.1.0,<2.0.0)
+Requires-Dist: easyocr (>=1.7,<2.0) ; extra == "easyocr" or extra == "ocr"
+Requires-Dist: filetype (>=1.2.0,<2.0.0)
+Requires-Dist: huggingface_hub (>=0.23,<1)
+Requires-Dist: pydantic (>=2.0.0,<3.0.0)
+Requires-Dist: pydantic-settings (>=2.3.0,<3.0.0)
+Requires-Dist: pypdfium2 (>=4.30.0,<5.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Project-URL: Repository, https://github.com/DS4SD/docling
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <a href="https://github.com/ds4sd/docling">
+    <img loading="lazy" alt="Docling" src="https://github.com/DS4SD/docling/raw/main/logo.png" width="150" />
+  </a>
+</p>
+
+# Docling
+
+[![PyPI version](https://img.shields.io/pypi/v/docling)](https://pypi.org/project/docling/)
+![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
+[![License MIT](https://img.shields.io/github/license/DS4SD/docling)](https://opensource.org/licenses/MIT)
+
+Docling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.
+
+## Features
+* ⚡ Converts any PDF document to JSON or Markdown format, stable and lightning fast
+* 📑 Understands detailed page layout, reading order and recovers table structures
+* 📝 Extracts metadata from the document, such as title, authors, references and language
+* 🔍 Optionally applies OCR (use with scanned PDFs)
+
+## Installation
+
+To use Docling, simply install `docling` from your package manager, e.g. pip:
+```bash
+pip install docling
+```
+
+> [!NOTE]
+> Works on macOS and Linux environments. Windows platforms are currently not tested.
+
+### Development setup
+
+To develop for Docling, you need Python 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:
+```bash
+poetry install --all-extras
+```
+
+## Usage
+
+### Convert a single document
+
+To convert invidual PDF documents, use `convert_single()`, for example:
+```python
+from docling.document_converter import DocumentConverter
+
+source = "https://arxiv.org/pdf/2206.01062"  # PDF path or URL
+converter = DocumentConverter()
+doc = converter.convert_single(source)
+print(doc.export_to_markdown())  # output: "## DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis [...]"
+```
+
+### Convert a batch of documents
+
+For an example of batch-converting documents, see [convert.py](https://github.com/DS4SD/docling/blob/main/examples/convert.py).
+
+From a local repo clone, you can run it with:
+
+```
+python examples/convert.py
+```
+The output of the above command will be written to `./scratch`.
+
+### Adjust pipeline features
+
+#### Control pipeline options
+
+You can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter`:
+```python
+doc_converter = DocumentConverter(
+    artifacts_path=artifacts_path,
+    pipeline_options=PipelineOptions(
+        do_table_structure=False,  # controls if table structure is recovered
+        do_ocr=True,  # controls if OCR is applied (ignores programmatic content)
+    ),
+)
+```
+
+#### Control table extraction options
+
+You can control if table structure recognition should map the recognized structure back to PDF cells (default) or use text cells from the structure prediction itself.
+This can improve output quality if you find that multiple columns in extracted tables are erroneously merged into one.
+
+
+```python
+pipeline_options = PipelineOptions(do_table_structure=True)
+pipeline_options.table_structure_options.do_cell_matching = False  # uses text cells predicted from table structure model
+
+doc_converter = DocumentConverter(
+    artifacts_path=artifacts_path,
+    pipeline_options=pipeline_options,
+)
+```
+
+### Impose limits on the document size
+
+You can limit the file size and number of pages which should be allowed to process per document:
+```python
+conv_input = DocumentConversionInput.from_paths(
+    paths=[Path("./test/data/2206.01062.pdf")],
+    limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)
+)
+```
+
+### Convert from binary PDF streams
+
+You can convert PDFs from a binary stream instead of from the filesystem as follows:
+```python
+buf = BytesIO(your_binary_stream)
+docs = [DocumentStream(filename="my_doc.pdf", stream=buf)]
+conv_input = DocumentConversionInput.from_streams(docs)
+converted_docs = doc_converter.convert(conv_input)
+```
+### Limit resource usage
+
+You can limit the CPU threads used by Docling by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.
+
+
+## Contributing
+
+Please read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.
+
+
+## References
+
+If you use Docling in your projects, please consider citing the following:
+
+```bib
+@software{Docling,
+author = {Deep Search Team},
+month = {7},
+title = {{Docling}},
+url = {https://github.com/DS4SD/docling},
+version = {main},
+year = {2024}
+}
+```
+
+## License
+
+The Docling codebase is under MIT license.
+For individual model usage, please refer to the model licenses found in the original packages.
+

docling-1.1.2/README.md

@@ -0,0 +1,146 @@
+<p align="center">
+  <a href="https://github.com/ds4sd/docling">
+    <img loading="lazy" alt="Docling" src="https://github.com/DS4SD/docling/raw/main/logo.png" width="150" />
+  </a>
+</p>
+
+# Docling
+
+[![PyPI version](https://img.shields.io/pypi/v/docling)](https://pypi.org/project/docling/)
+![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
+[![License MIT](https://img.shields.io/github/license/DS4SD/docling)](https://opensource.org/licenses/MIT)
+
+Docling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.
+
+## Features
+* ⚡ Converts any PDF document to JSON or Markdown format, stable and lightning fast
+* 📑 Understands detailed page layout, reading order and recovers table structures
+* 📝 Extracts metadata from the document, such as title, authors, references and language
+* 🔍 Optionally applies OCR (use with scanned PDFs)
+
+## Installation
+
+To use Docling, simply install `docling` from your package manager, e.g. pip:
+```bash
+pip install docling
+```
+
+> [!NOTE]
+> Works on macOS and Linux environments. Windows platforms are currently not tested.
+
+### Development setup
+
+To develop for Docling, you need Python 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:
+```bash
+poetry install --all-extras
+```
+
+## Usage
+
+### Convert a single document
+
+To convert invidual PDF documents, use `convert_single()`, for example:
+```python
+from docling.document_converter import DocumentConverter
+
+source = "https://arxiv.org/pdf/2206.01062"  # PDF path or URL
+converter = DocumentConverter()
+doc = converter.convert_single(source)
+print(doc.export_to_markdown())  # output: "## DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis [...]"
+```
+
+### Convert a batch of documents
+
+For an example of batch-converting documents, see [convert.py](https://github.com/DS4SD/docling/blob/main/examples/convert.py).
+
+From a local repo clone, you can run it with:
+
+```
+python examples/convert.py
+```
+The output of the above command will be written to `./scratch`.
+
+### Adjust pipeline features
+
+#### Control pipeline options
+
+You can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter`:
+```python
+doc_converter = DocumentConverter(
+    artifacts_path=artifacts_path,
+    pipeline_options=PipelineOptions(
+        do_table_structure=False,  # controls if table structure is recovered
+        do_ocr=True,  # controls if OCR is applied (ignores programmatic content)
+    ),
+)
+```
+
+#### Control table extraction options
+
+You can control if table structure recognition should map the recognized structure back to PDF cells (default) or use text cells from the structure prediction itself.
+This can improve output quality if you find that multiple columns in extracted tables are erroneously merged into one.
+
+
+```python
+pipeline_options = PipelineOptions(do_table_structure=True)
+pipeline_options.table_structure_options.do_cell_matching = False  # uses text cells predicted from table structure model
+
+doc_converter = DocumentConverter(
+    artifacts_path=artifacts_path,
+    pipeline_options=pipeline_options,
+)
+```
+
+### Impose limits on the document size
+
+You can limit the file size and number of pages which should be allowed to process per document:
+```python
+conv_input = DocumentConversionInput.from_paths(
+    paths=[Path("./test/data/2206.01062.pdf")],
+    limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)
+)
+```
+
+### Convert from binary PDF streams
+
+You can convert PDFs from a binary stream instead of from the filesystem as follows:
+```python
+buf = BytesIO(your_binary_stream)
+docs = [DocumentStream(filename="my_doc.pdf", stream=buf)]
+conv_input = DocumentConversionInput.from_streams(docs)
+converted_docs = doc_converter.convert(conv_input)
+```
+### Limit resource usage
+
+You can limit the CPU threads used by Docling by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.
+
+
+## Contributing
+
+Please read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.
+
+
+## References
+
+If you use Docling in your projects, please consider citing the following:
+
+```bib
+@software{Docling,
+author = {Deep Search Team},
+month = {7},
+title = {{Docling}},
+url = {https://github.com/DS4SD/docling},
+version = {main},
+year = {2024}
+}
+```
+
+## License
+
+The Docling codebase is under MIT license.
+For individual model usage, please refer to the model licenses found in the original packages.

{docling-0.1.2 → docling-1.1.2}/docling/backend/pypdfium2_backend.py

@@ -201,13 +201,7 @@ class PyPdfiumPageBackend(PdfPageBackend):
 class PyPdfiumDocumentBackend(PdfDocumentBackend):
     def __init__(self, path_or_stream: Iterable[Union[BytesIO, Path]]):
         super().__init__(path_or_stream)
-
-        if isinstance(path_or_stream, Path):
-            self._pdoc = pdfium.PdfDocument(path_or_stream)
-        elif isinstance(path_or_stream, BytesIO):
-            self._pdoc = pdfium.PdfDocument(
-                path_or_stream
-            )  # TODO Fix me, won't accept bytes.
+        self._pdoc = pdfium.PdfDocument(path_or_stream)
 
     def page_count(self) -> int:
         return len(self._pdoc)

{docling-0.1.2 → docling-1.1.2}/docling/datamodel/base_models.py

@@ -1,3 +1,4 @@
+import copy
 from enum import Enum, auto
 from io import BytesIO
 from typing import Any, Dict, List, Optional, Tuple, Union
@@ -47,6 +48,15 @@ class BoundingBox(BaseModel):
     def height(self):
         return abs(self.t - self.b)
 
+    def scaled(self, scale: float) -> "BoundingBox":
+        out_bbox = copy.deepcopy(self)
+        out_bbox.l *= scale
+        out_bbox.r *= scale
+        out_bbox.t *= scale
+        out_bbox.b *= scale
+
+        return out_bbox
+
     def as_tuple(self):
         if self.coord_origin == CoordOrigin.TOPLEFT:
             return (self.l, self.t, self.r, self.b)
@@ -180,8 +190,7 @@ class TableStructurePrediction(BaseModel):
     table_map: Dict[int, TableElement] = {}
 
 
-class TextElement(BasePageElement):
-    ...
+class TextElement(BasePageElement): ...
 
 
 class FigureData(BaseModel):
@@ -242,6 +251,17 @@ class DocumentStream(BaseModel):
     stream: BytesIO
 
 
+class TableStructureOptions(BaseModel):
+    do_cell_matching: bool = (
+        True
+        # True:  Matches predictions back to PDF cells. Can break table output if PDF cells
+        #        are merged across table columns.
+        # False: Let table structure model define the text cells, ignore PDF cells.
+    )
+
+
 class PipelineOptions(BaseModel):
-    do_table_structure: bool = True
-    do_ocr: bool = False
+    do_table_structure: bool = True  # True: perform table structure extraction
+    do_ocr: bool = False  # True: perform OCR, replace programmatic PDF text
+
+    table_structure_options: TableStructureOptions = TableStructureOptions()

{docling-0.1.2 → docling-1.1.2}/docling/datamodel/document.py

@@ -3,7 +3,6 @@ from io import BytesIO
 from pathlib import Path, PurePath
 from typing import ClassVar, Dict, Iterable, List, Optional, Type, Union
 
-from deepsearch.documents.core.export import export_to_markdown
 from docling_core.types import BaseCell, BaseText
 from docling_core.types import BoundingBox as DsBoundingBox
 from docling_core.types import Document as DsDocument
@@ -117,16 +116,16 @@ class ConvertedDocument(BaseModel):
     errors: List[Dict] = []  # structure to keep errors
 
     pages: List[Page] = []
-    assembled: AssembledUnit = None
+    assembled: Optional[AssembledUnit] = None
 
-    output: DsDocument = None
+    output: Optional[DsDocument] = None
 
     def to_ds_document(self) -> DsDocument:
         title = ""
         desc = DsDocumentDescription(logs=[])
 
         page_hashes = [
-            PageReference(hash=p.page_hash, page=p.page_no, model="default")
+            PageReference(hash=p.page_hash, page=p.page_no + 1, model="default")
             for p in self.pages
         ]
 
@@ -160,7 +159,7 @@ class ConvertedDocument(BaseModel):
                         prov=[
                             Prov(
                                 bbox=target_bbox,
-                                page=element.page_no,
+                                page=element.page_no + 1,
                                 span=[0, len(element.text)],
                             )
                         ],
@@ -243,7 +242,7 @@ class ConvertedDocument(BaseModel):
                         prov=[
                             Prov(
                                 bbox=target_bbox,
-                                page=element.page_no,
+                                page=element.page_no + 1,
                                 span=[0, 0],
                             )
                         ],
@@ -265,7 +264,7 @@ class ConvertedDocument(BaseModel):
                         prov=[
                             Prov(
                                 bbox=target_bbox,
-                                page=element.page_no,
+                                page=element.page_no + 1,
                                 span=[0, 0],
                             )
                         ],
@@ -275,7 +274,7 @@ class ConvertedDocument(BaseModel):
                 )
 
         page_dimensions = [
-            PageDimensions(page=p.page_no, height=p.size.height, width=p.size.width)
+            PageDimensions(page=p.page_no + 1, height=p.size.height, width=p.size.width)
             for p in self.pages
         ]
 
@@ -299,9 +298,7 @@ class ConvertedDocument(BaseModel):
 
     def render_as_markdown(self):
         if self.output:
-            return export_to_markdown(
-                self.output.model_dump(by_alias=True, exclude_none=True)
-            )
+            return self.output.export_to_markdown()
         else:
             return ""
 

{docling-0.1.2 → docling-1.1.2}/docling/document_converter.py

@@ -1,11 +1,15 @@
 import functools
 import logging
+import tempfile
 import time
 import traceback
 from pathlib import Path
 from typing import Iterable, Optional, Type, Union
 
+import requests
+from docling_core.types import Document
 from PIL import ImageDraw
+from pydantic import AnyHttpUrl, TypeAdapter, ValidationError
 
 from docling.backend.abstract_backend import PdfDocumentBackend
 from docling.datamodel.base_models import (
@@ -32,6 +36,7 @@ _log = logging.getLogger(__name__)
 class DocumentConverter:
     _layout_model_path = "model_artifacts/layout/beehive_v0.0.5"
     _table_model_path = "model_artifacts/tableformer"
+    _default_download_filename = "file.pdf"
 
     def __init__(
         self,
@@ -80,6 +85,57 @@ class DocumentConverter:
             # Note: Pdfium backend is not thread-safe, thread pool usage was disabled.
             yield from map(self.process_document, input_batch)
 
+    def convert_single(self, source: Path | AnyHttpUrl | str) -> Document:
+        """Convert a single document.
+
+        Args:
+            source (Path | AnyHttpUrl | str): The PDF input source. Can be a path or URL.
+
+        Raises:
+            ValueError: If source is of unexpected type.
+            RuntimeError: If conversion fails.
+
+        Returns:
+            Document: The converted document object.
+        """
+        with tempfile.TemporaryDirectory() as temp_dir:
+            try:
+                http_url: AnyHttpUrl = TypeAdapter(AnyHttpUrl).validate_python(source)
+                res = requests.get(http_url, stream=True)
+                res.raise_for_status()
+                fname = None
+                # try to get filename from response header
+                if cont_disp := res.headers.get("Content-Disposition"):
+                    for par in cont_disp.strip().split(";"):
+                        # currently only handling directive "filename" (not "*filename")
+                        if (split := par.split("=")) and split[0].strip() == "filename":
+                            fname = "=".join(split[1:]).strip().strip("'\"") or None
+                            break
+                # otherwise, use name from URL:
+                if fname is None:
+                    fname = Path(http_url.path).name or self._default_download_filename
+                local_path = Path(temp_dir) / fname
+                with open(local_path, "wb") as f:
+                    for chunk in res.iter_content(chunk_size=1024):  # using 1-KB chunks
+                        f.write(chunk)
+            except ValidationError:
+                try:
+                    local_path = TypeAdapter(Path).validate_python(source)
+                except ValidationError:
+                    raise ValueError(
+                        f"Unexpected file path type encountered: {type(source)}"
+                    )
+            conv_inp = DocumentConversionInput.from_paths(paths=[local_path])
+            converted_docs_iter = self.convert(conv_inp)
+            converted_doc: ConvertedDocument = next(converted_docs_iter)
+        if converted_doc.status not in {
+            ConversionStatus.SUCCESS,
+            ConversionStatus.SUCCESS_WITH_ERRORS,
+        }:
+            raise RuntimeError(f"Conversion failed with status: {converted_doc.status}")
+        doc = converted_doc.to_ds_document()
+        return doc
+
     def process_document(self, in_doc: InputDocument) -> ConvertedDocument:
         start_doc_time = time.time()
         converted_doc = ConvertedDocument(input=in_doc)

{docling-0.1.2 → docling-1.1.2}/docling/models/page_assemble_model.py

@@ -19,18 +19,6 @@ class PageAssembleModel:
     def __init__(self, config):
         self.config = config
 
-        # self.line_wrap_pattern = re.compile(r'(?<=[^\W_])- \n(?=\w)')
-
-    # def sanitize_text_poor(self, lines):
-    #     text = '\n'.join(lines)
-    #
-    #     # treat line wraps.
-    #     sanitized_text = self.line_wrap_pattern.sub('', text)
-    #
-    #     sanitized_text = sanitized_text.replace('\n', ' ')
-    #
-    #     return sanitized_text
-
     def sanitize_text(self, lines):
         if len(lines) <= 1:
             return " ".join(lines)

{docling-0.1.2 → docling-1.1.2}/docling/models/table_structure_model.py

@@ -1,7 +1,10 @@
-from typing import Iterable
+import copy
+import random
+from typing import Iterable, List
 
 import numpy
 from docling_ibm_models.tableformer.data_management.tf_predictor import TFPredictor
+from PIL import ImageDraw
 
 from docling.datamodel.base_models import (
     BoundingBox,
@@ -28,6 +31,21 @@ class TableStructureModel:
             self.tm_model_type = self.tm_config["model"]["type"]
 
             self.tf_predictor = TFPredictor(self.tm_config)
+            self.scale = 2.0  # Scale up table input images to 144 dpi
+
+    def draw_table_and_cells(self, page: Page, tbl_list: List[TableElement]):
+        image = page._backend.get_page_image()
+        draw = ImageDraw.Draw(image)
+
+        for table_element in tbl_list:
+            x0, y0, x1, y1 = table_element.cluster.bbox.as_tuple()
+            draw.rectangle([(x0, y0), (x1, y1)], outline="red")
+
+            for tc in table_element.table_cells:
+                x0, y0, x1, y1 = tc.bbox.as_tuple()
+                draw.rectangle([(x0, y0), (x1, y1)], outline="blue")
+
+        image.show()
 
     def __call__(self, page_batch: Iterable[Page]) -> Iterable[Page]:
 
@@ -36,16 +54,17 @@ class TableStructureModel:
             return
 
         for page in page_batch:
+
             page.predictions.tablestructure = TableStructurePrediction()  # dummy
 
             in_tables = [
                 (
                     cluster,
                     [
-                        round(cluster.bbox.l),
-                        round(cluster.bbox.t),
-                        round(cluster.bbox.r),
-                        round(cluster.bbox.b),
+                        round(cluster.bbox.l) * self.scale,
+                        round(cluster.bbox.t) * self.scale,
+                        round(cluster.bbox.r) * self.scale,
+                        round(cluster.bbox.b) * self.scale,
                     ],
                 )
                 for cluster in page.predictions.layout.clusters
@@ -65,20 +84,29 @@ class TableStructureModel:
                         ):
                             # Only allow non empty stings (spaces) into the cells of a table
                             if len(c.text.strip()) > 0:
-                                tokens.append(c.model_dump())
+                                new_cell = copy.deepcopy(c)
+                                new_cell.bbox = new_cell.bbox.scaled(scale=self.scale)
+
+                                tokens.append(new_cell.model_dump())
 
-            iocr_page = {
-                "image": numpy.asarray(page.image),
+            page_input = {
                 "tokens": tokens,
-                "width": page.size.width,
-                "height": page.size.height,
+                "width": page.size.width * self.scale,
+                "height": page.size.height * self.scale,
             }
+            # add image to page input.
+            if self.scale == 1.0:
+                page_input["image"] = numpy.asarray(page.image)
+            else:  # render new page image on the fly at desired scale
+                page_input["image"] = numpy.asarray(
+                    page._backend.get_page_image(scale=self.scale)
+                )
 
             table_clusters, table_bboxes = zip(*in_tables)
 
             if len(table_bboxes):
                 tf_output = self.tf_predictor.multi_table_predict(
-                    iocr_page, table_bboxes, do_matching=self.do_cell_matching
+                    page_input, table_bboxes, do_matching=self.do_cell_matching
                 )
 
                 for table_cluster, table_out in zip(table_clusters, tf_output):
@@ -86,11 +114,15 @@ class TableStructureModel:
                     for element in table_out["tf_responses"]:
 
                         if not self.do_cell_matching:
-                            the_bbox = BoundingBox.model_validate(element["bbox"])
+                            the_bbox = BoundingBox.model_validate(
+                                element["bbox"]
+                            ).scaled(1 / self.scale)
                             text_piece = page._backend.get_text_in_rect(the_bbox)
                             element["bbox"]["token"] = text_piece
 
                         tc = TableCell.model_validate(element)
+                        if self.do_cell_matching:
+                            tc.bbox = tc.bbox.scaled(1 / self.scale)
                         table_cells.append(tc)
 
                     # Retrieving cols/rows, after post processing:
@@ -111,4 +143,7 @@ class TableStructureModel:
 
                     page.predictions.tablestructure.table_map[table_cluster.id] = tbl
 
+                # For debugging purposes:
+                # self.draw_table_and_cells(page, page.predictions.tablestructure.table_map.values())
+
             yield page

{docling-0.1.2 → docling-1.1.2}/docling/pipeline/standard_model_pipeline.py

@@ -34,7 +34,7 @@ class StandardModelPipeline(BaseModelPipeline):
                     "artifacts_path": artifacts_path
                     / StandardModelPipeline._table_model_path,
                     "enabled": pipeline_options.do_table_structure,
-                    "do_cell_matching": False,
+                    "do_cell_matching": pipeline_options.table_structure_options.do_cell_matching,
                 }
             ),
         ]

{docling-0.1.2 → docling-1.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "docling"
-version = "0.1.2"
+version = "1.1.2"  # DO NOT EDIT, updated automatically
 description = "Docling PDF conversion package"
 authors = ["Christoph Auer <cau@zurich.ibm.com>", "Michele Dolfi <dol@zurich.ibm.com>", "Maxim Lysak <mly@zurich.ibm.com>", "Nikos Livathinos <nli@zurich.ibm.com>", "Ahmed Nassar <ahn@zurich.ibm.com>", "Peter Staar <taa@zurich.ibm.com>"]
 license = "MIT"
@@ -21,19 +21,17 @@ keywords= ["docling", "convert", "document", "pdf", "layout model", "segmentatio
 packages = [{include = "docling"}]
 
 [tool.poetry.dependencies]
-python = "^3.11"
+python = "^3.10"
 pydantic = "^2.0.0"
-docling-core = "^0.2.0"
-docling-ibm-models = "^0.2.0"
-deepsearch-glm = ">=0.18.4,<1"
-deepsearch-toolkit = ">=0.47.0,<1"
+docling-core = "^1.1.2"
+docling-ibm-models = "^1.1.0"
+deepsearch-glm = ">=0.19.0,<1"
 filetype = "^1.2.0"
 pypdfium2 = "^4.30.0"
 pydantic-settings = "^2.3.0"
 huggingface_hub = ">=0.23,<1"
-
-[tool.poetry.group.ocr.dependencies]
-easyocr = "^1.7"
+requests = "^2.32.3"
+easyocr = { version = "^1.7", optional = true }
 
 [tool.poetry.group.dev.dependencies]
 black = {extras = ["jupyter"], version = "^24.4.2"}
@@ -49,13 +47,17 @@ types-requests = "^2.31.0.2"
 flake8-pyproject = "^1.2.3"
 pylint = "^2.17.5"
 
+[tool.poetry.extras]
+easyocr = ["easyocr"]
+ocr = ["easyocr"]
+
 [build-system]
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"
 
 [tool.black]
 line-length = 88
-target-version = ["py311"]
+target-version = ["py310"]
 include = '\.pyi?$'
 
 [tool.isort]
@@ -67,8 +69,21 @@ py_version=311
 pretty = true
 # strict = true
 no_implicit_optional = true
-python_version = "3.11"
+python_version = "3.10"
 
 [tool.flake8]
 max-line-length = 88
 extend-ignore = ["E203", "E501"]
+
+[tool.semantic_release]
+# for default values check:
+# https://github.com/python-semantic-release/python-semantic-release/blob/v7.32.2/semantic_release/defaults.cfg
+
+version_source = "tag_only"
+branch = "main"
+
+# configure types which should trigger minor and patch version bumps respectively
+# (note that they must be a subset of the configured allowed types):
+parser_angular_allowed_types = "build,chore,ci,docs,feat,fix,perf,style,refactor,test"
+parser_angular_minor_types = "feat"
+parser_angular_patch_types = "fix,perf"

docling-0.1.2/PKG-INFO

@@ -1,132 +0,0 @@
-Metadata-Version: 2.1
-Name: docling
-Version: 0.1.2
-Summary: Docling PDF conversion package
-Home-page: https://github.com/DS4SD/docling
-License: MIT
-Keywords: docling,convert,document,pdf,layout model,segmentation,table structure,table former
-Author: Christoph Auer
-Author-email: cau@zurich.ibm.com
-Requires-Python: >=3.11,<4.0
-Classifier: Development Status :: 5 - Production/Stable
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: MacOS :: MacOS X
-Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Dist: deepsearch-glm (>=0.18.4,<1)
-Requires-Dist: deepsearch-toolkit (>=0.47.0,<1)
-Requires-Dist: docling-core (>=0.2.0,<0.3.0)
-Requires-Dist: docling-ibm-models (>=0.2.0,<0.3.0)
-Requires-Dist: filetype (>=1.2.0,<2.0.0)
-Requires-Dist: huggingface_hub (>=0.23,<1)
-Requires-Dist: pydantic (>=2.0.0,<3.0.0)
-Requires-Dist: pydantic-settings (>=2.3.0,<3.0.0)
-Requires-Dist: pypdfium2 (>=4.30.0,<5.0.0)
-Project-URL: Repository, https://github.com/DS4SD/docling
-Description-Content-Type: text/markdown
-
-<p align="center">
-  <a href="https://github.com/ds4sd/docling"> <img loading="lazy" alt="Docling" src="https://github.com/DS4SD/docling/raw/main/logo.png" width="150" /> </a>
-</p>
-
-# Docling
-
-Docling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.
-
-## Features
-* ⚡ Converts any PDF document to JSON or Markdown format, stable and lightning fast
-* 📑 Understands detailed page layout, reading order and recovers table structures
-* 📝 Extracts metadata from the document, such as title, authors, references and language
-* 🔍 Optionally applies OCR (use with scanned PDFs)
-
-## Setup
-
-You need Python 3.11 and poetry. Install poetry from [here](https://python-poetry.org/docs/#installing-with-the-official-installer).
-
-Once you have `poetry` installed, create an environment and install the package:
-
-```bash
-poetry env use $(which python3.11)
-poetry shell
-poetry install
-```
-
-**Notes**:
-* Works on macOS and Linux environments. Windows platforms are currently not tested.
-
-
-## Usage
-
-For basic usage, see the [convert.py](https://github.com/DS4SD/docling/blob/main/examples/convert.py) example module. Run with:
-
-```
-python examples/convert.py
-```
-The output of the above command will be written to `./scratch`.
-
-### Enable or disable pipeline features
-
-You can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter` 
-```python
-doc_converter = DocumentConverter(
-    artifacts_path=artifacts_path,
-    pipeline_options=PipelineOptions(do_table_structure=False, # Controls if table structure is recovered. 
-                                     do_ocr=True), # Controls if OCR is applied (ignores programmatic content)
-)
-```
-
-### Impose limits on the document size
-
-You can limit the file size and number of pages which should be allowed to process per document.
-```python
-paths = [Path("./test/data/2206.01062.pdf")]
-
-input = DocumentConversionInput.from_paths(
-    paths, limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)
-)
-```
-
-### Convert from binary PDF streams 
-
-You can convert PDFs from a binary stream instead of from the filesystem as follows:
-```python
-buf = BytesIO(your_binary_stream)
-docs = [DocumentStream(filename="my_doc.pdf", stream=buf)]
-input = DocumentConversionInput.from_streams(docs)
-converted_docs = doc_converter.convert(input)
-```
-### Limit resource usage
-
-You can limit the CPU threads used by `docling` by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.
-
-
-## Contributing
-
-Please read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.
-
-
-## References
-
-If you use `Docling` in your projects, please consider citing the following:
-
-```bib
-@software{Docling,
-author = {Deep Search Team},
-month = {7},
-title = {{Docling}},
-url = {https://github.com/DS4SD/docling},
-version = {main},
-year = {2024}
-}
-```
-
-## License
-
-The `Docling` codebase is under MIT license.
-For individual model usage, please refer to the model licenses found in the original packages.
-

docling-0.1.2/README.md

@@ -1,99 +0,0 @@
-<p align="center">
-  <a href="https://github.com/ds4sd/docling"> <img loading="lazy" alt="Docling" src="https://github.com/DS4SD/docling/raw/main/logo.png" width="150" /> </a>
-</p>
-
-# Docling
-
-Docling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.
-
-## Features
-* ⚡ Converts any PDF document to JSON or Markdown format, stable and lightning fast
-* 📑 Understands detailed page layout, reading order and recovers table structures
-* 📝 Extracts metadata from the document, such as title, authors, references and language
-* 🔍 Optionally applies OCR (use with scanned PDFs)
-
-## Setup
-
-You need Python 3.11 and poetry. Install poetry from [here](https://python-poetry.org/docs/#installing-with-the-official-installer).
-
-Once you have `poetry` installed, create an environment and install the package:
-
-```bash
-poetry env use $(which python3.11)
-poetry shell
-poetry install
-```
-
-**Notes**:
-* Works on macOS and Linux environments. Windows platforms are currently not tested.
-
-
-## Usage
-
-For basic usage, see the [convert.py](https://github.com/DS4SD/docling/blob/main/examples/convert.py) example module. Run with:
-
-```
-python examples/convert.py
-```
-The output of the above command will be written to `./scratch`.
-
-### Enable or disable pipeline features
-
-You can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter` 
-```python
-doc_converter = DocumentConverter(
-    artifacts_path=artifacts_path,
-    pipeline_options=PipelineOptions(do_table_structure=False, # Controls if table structure is recovered. 
-                                     do_ocr=True), # Controls if OCR is applied (ignores programmatic content)
-)
-```
-
-### Impose limits on the document size
-
-You can limit the file size and number of pages which should be allowed to process per document.
-```python
-paths = [Path("./test/data/2206.01062.pdf")]
-
-input = DocumentConversionInput.from_paths(
-    paths, limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)
-)
-```
-
-### Convert from binary PDF streams 
-
-You can convert PDFs from a binary stream instead of from the filesystem as follows:
-```python
-buf = BytesIO(your_binary_stream)
-docs = [DocumentStream(filename="my_doc.pdf", stream=buf)]
-input = DocumentConversionInput.from_streams(docs)
-converted_docs = doc_converter.convert(input)
-```
-### Limit resource usage
-
-You can limit the CPU threads used by `docling` by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.
-
-
-## Contributing
-
-Please read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.
-
-
-## References
-
-If you use `Docling` in your projects, please consider citing the following:
-
-```bib
-@software{Docling,
-author = {Deep Search Team},
-month = {7},
-title = {{Docling}},
-url = {https://github.com/DS4SD/docling},
-version = {main},
-year = {2024}
-}
-```
-
-## License
-
-The `Docling` codebase is under MIT license.
-For individual model usage, please refer to the model licenses found in the original packages.