docling-core 1.7.2__tar.gz → 2.0.1__tar.gz

{docling_core-1.7.2 → docling_core-2.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: docling-core
-Version: 1.7.2
+Version: 2.0.1
 Summary: A python library to define and validate data types in Docling.
 Home-page: https://ds4sd.github.io/
 License: MIT
@@ -25,10 +25,10 @@ Classifier: Topic :: Database
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
-Requires-Dist: json-schema-for-humans (>=1.0.0,<2.0.0)
 Requires-Dist: jsonref (>=1.1.0,<2.0.0)
 Requires-Dist: jsonschema (>=4.16.0,<5.0.0)
 Requires-Dist: pandas (>=2.1.4,<3.0.0)
+Requires-Dist: pillow (>=10.3.0,<11.0.0)
 Requires-Dist: pydantic (>=2.6.0,<3.0.0)
 Requires-Dist: tabulate (>=0.9.0,<0.10.0)
 Project-URL: Repository, https://github.com/DS4SD/docling-core
@@ -37,7 +37,7 @@ Description-Content-Type: text/markdown
 # Docling Core
 
 [![PyPI version](https://img.shields.io/pypi/v/docling-core)](https://pypi.org/project/docling-core/)
-![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
+![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%20%203.11%20%7C%203.12%20%7C%203.13-blue)
 [![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
@@ -57,7 +57,7 @@ pip install docling-core
 
 ### Development setup
 
-To develop for Docling Core, you need Python 3.9 / 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:
+To develop for Docling Core, you need Python 3.9 / 3.10 / 3.11 / 3.12 / 3.13 and Poetry. You can then install from your local clone's root dir:
 ```bash
 poetry install
 ```
@@ -72,37 +72,37 @@ poetry run pytest test
 - You can validate your JSON objects using the pydantic class definition.
 
   ```py
-  from docling_core.types import Document
+  from docling_core.types import DoclingDocument
 
   data_dict = {...}  # here the object you want to validate, as a dictionary
-  Document.model_validate(data_dict)
+  DoclingDocument.model_validate(data_dict)
 
   data_str = {...}  # here the object as a JSON string
-  Document.model_validate_json(data_str)
+  DoclingDocument.model_validate_json(data_str)
   ```
 
-- You can generate the JSON schema of a model with the script `ds_generate_jsonschema`.
+- You can generate the JSON schema of a model with the script `generate_jsonschema`.
 
   ```py
-  # for the `Document` type
-  ds_generate_jsonschema Document
+  # for the `DoclingDocument` type
+  generate_jsonschema DoclingDocument
 
   # for the use `Record` type
-  ds_generate_jsonschema Record
+  generate_jsonschema Record
   ```
 
 ## Documentation
 
-Docling supports 3 main data types:
+Docling Core contains 3 top-level data types:
 
-- **Document** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
-  The Document type also models the metadata that may be attached to the converted document.
-  Check [Document](docs/Document.md) for the full JSON schema. 
+- **DoclingDocument** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
+  The DoclingDocument type also models the metadata that may be attached to the converted document.
+  Check [DoclingDocument](docs/DoclingDocument.json) for the full JSON schema.
 - **Record** for structured database records, centered on an entity or _subject_ that is provided with a list of attributes.
   Related to records, the statements can represent annotations on text by Natural Language Processing (NLP) tools.
-  Check [Record](docs/Record.md) for the full JSON schema. 
+  Check [Record](docs/Record.json) for the full JSON schema.
 - **Generic** for any data representation, ensuring minimal configuration and maximum flexibility.
-  Check [Generic](docs/Generic.md) for the full JSON schema. 
+  Check [Generic](docs/Generic.json) for the full JSON schema.
 
 The data schemas are defined using [pydantic](https://pydantic-docs.helpmanual.io/) models, which provide built-in processes to support the creation of data that adhere to those models.
 

{docling_core-1.7.2 → docling_core-2.0.1}/README.md

@@ -1,7 +1,7 @@
 # Docling Core
 
 [![PyPI version](https://img.shields.io/pypi/v/docling-core)](https://pypi.org/project/docling-core/)
-![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
+![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%20%203.11%20%7C%203.12%20%7C%203.13-blue)
 [![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
@@ -21,7 +21,7 @@ pip install docling-core
 
 ### Development setup
 
-To develop for Docling Core, you need Python 3.9 / 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:
+To develop for Docling Core, you need Python 3.9 / 3.10 / 3.11 / 3.12 / 3.13 and Poetry. You can then install from your local clone's root dir:
 ```bash
 poetry install
 ```
@@ -36,37 +36,37 @@ poetry run pytest test
 - You can validate your JSON objects using the pydantic class definition.
 
   ```py
-  from docling_core.types import Document
+  from docling_core.types import DoclingDocument
 
   data_dict = {...}  # here the object you want to validate, as a dictionary
-  Document.model_validate(data_dict)
+  DoclingDocument.model_validate(data_dict)
 
   data_str = {...}  # here the object as a JSON string
-  Document.model_validate_json(data_str)
+  DoclingDocument.model_validate_json(data_str)
   ```
 
-- You can generate the JSON schema of a model with the script `ds_generate_jsonschema`.
+- You can generate the JSON schema of a model with the script `generate_jsonschema`.
 
   ```py
-  # for the `Document` type
-  ds_generate_jsonschema Document
+  # for the `DoclingDocument` type
+  generate_jsonschema DoclingDocument
 
   # for the use `Record` type
-  ds_generate_jsonschema Record
+  generate_jsonschema Record
   ```
 
 ## Documentation
 
-Docling supports 3 main data types:
+Docling Core contains 3 top-level data types:
 
-- **Document** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
-  The Document type also models the metadata that may be attached to the converted document.
-  Check [Document](docs/Document.md) for the full JSON schema. 
+- **DoclingDocument** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
+  The DoclingDocument type also models the metadata that may be attached to the converted document.
+  Check [DoclingDocument](docs/DoclingDocument.json) for the full JSON schema.
 - **Record** for structured database records, centered on an entity or _subject_ that is provided with a list of attributes.
   Related to records, the statements can represent annotations on text by Natural Language Processing (NLP) tools.
-  Check [Record](docs/Record.md) for the full JSON schema. 
+  Check [Record](docs/Record.json) for the full JSON schema.
 - **Generic** for any data representation, ensuring minimal configuration and maximum flexibility.
-  Check [Generic](docs/Generic.md) for the full JSON schema. 
+  Check [Generic](docs/Generic.json) for the full JSON schema.
 
 The data schemas are defined using [pydantic](https://pydantic-docs.helpmanual.io/) models, which provide built-in processes to support the creation of data that adhere to those models.
 

{docling_core-1.7.2 → docling_core-2.0.1}/docling_core/transforms/chunker/__init__.py

@@ -5,11 +5,5 @@
 
 """Define the chunker types."""
 
-from docling_core.transforms.chunker.base import (  # noqa
-    BaseChunker,
-    Chunk,
-    ChunkWithMetadata,
-)
-from docling_core.transforms.chunker.hierarchical_chunker import (  # noqa
-    HierarchicalChunker,
-)
+from docling_core.transforms.chunker.base import BaseChunk, BaseChunker, BaseMeta
+from docling_core.transforms.chunker.hierarchical_chunker import HierarchicalChunker

docling_core-2.0.1/docling_core/transforms/chunker/base.py

@@ -0,0 +1,61 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: MIT
+#
+
+"""Define base classes for chunking."""
+from abc import ABC, abstractmethod
+from typing import Any, ClassVar, Iterator
+
+from pydantic import BaseModel
+
+from docling_core.types.doc import DoclingDocument as DLDocument
+
+
+class BaseMeta(BaseModel):
+    """Metadata base class."""
+
+    excluded_embed: ClassVar[list[str]] = []
+    excluded_llm: ClassVar[list[str]] = []
+
+    def export_json_dict(self) -> dict[str, Any]:
+        """Helper method for exporting non-None keys to JSON mode.
+
+        Returns:
+            dict[str, Any]: The exported dictionary.
+        """
+        return self.model_dump(mode="json", by_alias=True, exclude_none=True)
+
+
+class BaseChunk(BaseModel):
+    """Chunk base class."""
+
+    text: str
+    meta: BaseMeta
+
+    def export_json_dict(self) -> dict[str, Any]:
+        """Helper method for exporting non-None keys to JSON mode.
+
+        Returns:
+            dict[str, Any]: The exported dictionary.
+        """
+        return self.model_dump(mode="json", by_alias=True, exclude_none=True)
+
+
+class BaseChunker(BaseModel, ABC):
+    """Chunker base class."""
+
+    @abstractmethod
+    def chunk(self, dl_doc: DLDocument, **kwargs) -> Iterator[BaseChunk]:
+        """Chunk the provided document.
+
+        Args:
+            dl_doc (DLDocument): document to chunk
+
+        Raises:
+            NotImplementedError: in this abstract implementation
+
+        Yields:
+            Iterator[BaseChunk]: iterator over extracted chunks
+        """
+        raise NotImplementedError()

docling_core-2.0.1/docling_core/transforms/chunker/hierarchical_chunker.py

@@ -0,0 +1,186 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: MIT
+#
+
+"""Chunker implementation leveraging the document structure."""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, ClassVar, Iterator, Optional
+
+from pandas import DataFrame
+from pydantic import Field
+
+from docling_core.transforms.chunker import BaseChunk, BaseChunker, BaseMeta
+from docling_core.types.doc import DoclingDocument as DLDocument
+from docling_core.types.doc.document import (
+    DocItem,
+    LevelNumber,
+    ListItem,
+    SectionHeaderItem,
+    TableItem,
+    TextItem,
+)
+from docling_core.types.doc.labels import DocItemLabel
+
+_KEY_DOC_ITEMS = "doc_items"
+_KEY_HEADINGS = "headings"
+_KEY_CAPTIONS = "captions"
+
+_logger = logging.getLogger(__name__)
+
+
+class DocMeta(BaseMeta):
+    """Data model for Hierarchical Chunker metadata."""
+
+    doc_items: list[DocItem] = Field(
+        alias=_KEY_DOC_ITEMS,
+        min_length=1,
+    )
+    headings: Optional[list[str]] = Field(
+        default=None,
+        alias=_KEY_HEADINGS,
+        min_length=1,
+    )
+    captions: Optional[list[str]] = Field(
+        default=None,
+        alias=_KEY_CAPTIONS,
+        min_length=1,
+    )
+
+    excluded_embed: ClassVar[list[str]] = [_KEY_DOC_ITEMS]
+    excluded_llm: ClassVar[list[str]] = [_KEY_DOC_ITEMS]
+
+
+class DocChunk(BaseChunk):
+    """Data model for Hierarchical Chunker chunks."""
+
+    meta: DocMeta
+
+
+class HierarchicalChunker(BaseChunker):
+    r"""Chunker implementation leveraging the document layout.
+
+    Args:
+        merge_list_items (bool): Whether to merge successive list items.
+            Defaults to True.
+        delim (str): Delimiter to use for merging text. Defaults to "\n".
+    """
+
+    merge_list_items: bool = True
+    delim: str = "\n"
+
+    @classmethod
+    def _triplet_serialize(cls, table_df: DataFrame) -> str:
+
+        # copy header as first row and shift all rows by one
+        table_df.loc[-1] = table_df.columns  # type: ignore[call-overload]
+        table_df.index = table_df.index + 1
+        table_df = table_df.sort_index()
+
+        rows = [item.strip() for item in table_df.iloc[:, 0].to_list()]
+        cols = [item.strip() for item in table_df.iloc[0, :].to_list()]
+
+        nrows = table_df.shape[0]
+        ncols = table_df.shape[1]
+        texts = [
+            f"{rows[i]}, {cols[j]} = {str(table_df.iloc[i, j]).strip()}"
+            for i in range(1, nrows)
+            for j in range(1, ncols)
+        ]
+        output_text = ". ".join(texts)
+
+        return output_text
+
+    def chunk(self, dl_doc: DLDocument, **kwargs: Any) -> Iterator[BaseChunk]:
+        r"""Chunk the provided document.
+
+        Args:
+            dl_doc (DLDocument): document to chunk
+
+        Yields:
+            Iterator[Chunk]: iterator over extracted chunks
+        """
+        heading_by_level: dict[LevelNumber, str] = {}
+        list_items: list[TextItem] = []
+        for item, level in dl_doc.iterate_items():
+            captions = None
+            if isinstance(item, DocItem):
+
+                # first handle any merging needed
+                if self.merge_list_items:
+                    if isinstance(
+                        item, ListItem
+                    ) or (  # TODO remove when all captured as ListItem:
+                        isinstance(item, TextItem)
+                        and item.label == DocItemLabel.LIST_ITEM
+                    ):
+                        list_items.append(item)
+                        continue
+                    elif list_items:  # need to yield
+                        yield DocChunk(
+                            text=self.delim.join([i.text for i in list_items]),
+                            meta=DocMeta(
+                                doc_items=list_items,
+                                headings=[
+                                    heading_by_level[k]
+                                    for k in sorted(heading_by_level)
+                                ]
+                                or None,
+                            ),
+                        )
+                        list_items = []  # reset
+
+                if isinstance(
+                    item, SectionHeaderItem
+                ) or (  # TODO remove when all captured as SectionHeaderItem:
+                    isinstance(item, TextItem)
+                    and item.label == DocItemLabel.SECTION_HEADER
+                ):
+                    # TODO second branch not needed once cleanup above complete:
+                    level = item.level if isinstance(item, SectionHeaderItem) else 1
+                    heading_by_level[level] = item.text
+
+                    # remove headings of higher level as they just went out of scope
+                    keys_to_del = [k for k in heading_by_level if k > level]
+                    for k in keys_to_del:
+                        heading_by_level.pop(k, None)
+                    continue
+
+                if isinstance(item, TextItem) or (
+                    (not self.merge_list_items) and isinstance(item, ListItem)
+                ):
+                    text = item.text
+                elif isinstance(item, TableItem):
+                    table_df = item.export_to_dataframe()
+                    if table_df.shape[0] < 1 or table_df.shape[1] < 2:
+                        # at least two cols needed, as first column contains row headers
+                        continue
+                    text = self._triplet_serialize(table_df=table_df)
+                    captions = [
+                        c.text for c in [r.resolve(dl_doc) for r in item.captions]
+                    ] or None
+                else:
+                    continue
+                c = DocChunk(
+                    text=text,
+                    meta=DocMeta(
+                        doc_items=[item],
+                        headings=[heading_by_level[k] for k in sorted(heading_by_level)]
+                        or None,
+                        captions=captions,
+                    ),
+                )
+                yield c
+
+        if self.merge_list_items and list_items:  # need to yield
+            yield DocChunk(
+                text=self.delim.join([i.text for i in list_items]),
+                meta=DocMeta(
+                    doc_items=list_items,
+                    headings=[heading_by_level[k] for k in sorted(heading_by_level)]
+                    or None,
+                ),
+            )

docling_core-2.0.1/docling_core/types/__init__.py

@@ -0,0 +1,10 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: MIT
+#
+
+"""Define the main types."""
+
+from docling_core.types.doc.document import DoclingDocument
+from docling_core.types.gen.generic import Generic
+from docling_core.types.rec.record import Record

{docling_core-1.7.2/docling_core/types/experimental → docling_core-2.0.1/docling_core/types/doc}/__init__.py

@@ -7,9 +7,6 @@
 
 from .base import BoundingBox, CoordOrigin, Size
 from .document import (
-    BasePictureData,
-    BaseTableData,
-    DescriptionItem,
     DocItem,
     DoclingDocument,
     DocumentOrigin,
@@ -19,11 +16,15 @@ from .document import (
     KeyValueItem,
     NodeItem,
     PageItem,
+    PictureClassificationClass,
+    PictureClassificationData,
+    PictureDataType,
     PictureItem,
     ProvenanceItem,
     RefItem,
     SectionHeaderItem,
     TableCell,
+    TableData,
     TableItem,
     TextItem,
 )

{docling_core-1.7.2/docling_core/types/experimental → docling_core-2.0.1/docling_core/types/doc}/base.py

@@ -108,7 +108,10 @@ class BoundingBox(BaseModel):
 
     def area(self) -> float:
         """area."""
-        return (self.r - self.l) * (self.b - self.t)
+        area = (self.r - self.l) * (self.b - self.t)
+        if self.coord_origin == CoordOrigin.BOTTOMLEFT:
+            area = -area
+        return area
 
     def intersection_area_with(self, other: "BoundingBox") -> float:
         """intersection_area_with.