docling-core 0.0.1__tar.gz

docling_core-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 International Business Machines
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

docling_core-0.0.1/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.1
+Name: docling-core
+Version: 0.0.1
+Summary: A python library to define and validate data types in Docling.
+Home-page: https://ds4sd.github.io/
+License: MIT
+Keywords: docling,discovery,etl,information retrieval,analytics,database,database schema,schema,JSON
+Author: Cesar Berrospi Ramis
+Author-email: ceb@zurich.ibm.com
+Maintainer: Cesar Berrospi Ramis
+Maintainer-email: ceb@zurich.ibm.com
+Requires-Python: >=3.9,<4.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Dist: json-schema-for-humans (>=1.0.0,<2.0.0)
+Requires-Dist: jsonref (>=1.1.0,<2.0.0)
+Requires-Dist: jsonschema (>=4.16.0,<5.0.0)
+Requires-Dist: poetry (>=1.8.3,<2.0.0)
+Requires-Dist: pydantic (>=2.6.0,<3.0.0)
+Requires-Dist: pyproject-toml (>=0.0.10,<0.0.11)
+Project-URL: Repository, https://github.com/DS4SD/docling-core
+Description-Content-Type: text/markdown
+
+[![PyPI version](https://img.shields.io/pypi/v/docling-core)](https://pypi.org/project/docling-core/)
+![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11-blue)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
+[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
+[![License MIT](https://img.shields.io/github/license/ds4sd/deepsearch-toolkit)](https://opensource.org/licenses/MIT)
+
+# Docling Core
+
+Docling Core is a library that defines the data types in [Docling](https://ds4sd.github.io), leveraging pydantic models.
+
+## Installation
+
+Using [Poetry](https://python-poetry.org), create and activate a virtual environment.
+
+```
+poetry shell
+```
+
+Install the defined dependencies of the project.
+
+```
+poetry install
+```
+
+Test the installation running the pytest suite.
+
+```
+poetry run pytest test
+```
+
+## Basic Usage
+
+- You can validate your JSON objects using the pydantic class definition.
+
+  ```py
+  from docling_core.types import Document
+
+  data_dict = {...}  # here the object you want to validate, as a dictionary
+  Document.model_validate(data_dict)
+
+  data_str = {...}  # here the object as a JSON string
+  Document.model_validate_json(data_str)
+  ```
+
+- You can generate the JSON schema of a model with the script `ds_generate_jsonschema`.
+
+  ```py
+  # for the `Document` type
+  ds_generate_jsonschema Document
+
+  # for the use `Record` type
+  ds_generate_jsonschema Record
+  ```
+
+## Documentation
+
+Docling supports 3 main data types:
+
+- **Document** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
+  The Document type also models the metadata that may be attached to the converted document.
+  Check [Document](docs/Document.md) for the full JSON schema. 
+- **Record** for structured database records, centered on an entity or _subject_ that is provided with a list of attributes.
+  Related to records, the statements can represent annotations on text by Natural Language Processing (NLP) tools.
+  Check [Record](docs/Record.md) for the full JSON schema. 
+- **Generic** for any data representation, ensuring minimal configuration and maximum flexibility.
+  Check [Generic](docs/Generic.md) for the full JSON schema. 
+
+The data schemas are defined using [pydantic](https://pydantic-docs.helpmanual.io/) models, which provide built-in processes to support the creation of data that adhere to those models.
+
+## Contributing
+
+Please read [Contributing to Docling Core](./CONTRIBUTING.md) for details.
+
+## References
+
+If you use `Docling Core` in your projects, please consider citing the following:
+
+```bib
+@software{Docling,
+author = {Deep Search Team},
+month = {7},
+title = {{Docling}},
+url = {https://github.com/DS4SD/docling},
+version = {main},
+year = {2024}
+}
+```
+
+## License
+
+The `Docling Core` codebase is under MIT license.
+For individual model usage, please refer to the model licenses found in the original packages.
+

docling_core-0.0.1/README.md

@@ -0,0 +1,96 @@
+[![PyPI version](https://img.shields.io/pypi/v/docling-core)](https://pypi.org/project/docling-core/)
+![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11-blue)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
+[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
+[![License MIT](https://img.shields.io/github/license/ds4sd/deepsearch-toolkit)](https://opensource.org/licenses/MIT)
+
+# Docling Core
+
+Docling Core is a library that defines the data types in [Docling](https://ds4sd.github.io), leveraging pydantic models.
+
+## Installation
+
+Using [Poetry](https://python-poetry.org), create and activate a virtual environment.
+
+```
+poetry shell
+```
+
+Install the defined dependencies of the project.
+
+```
+poetry install
+```
+
+Test the installation running the pytest suite.
+
+```
+poetry run pytest test
+```
+
+## Basic Usage
+
+- You can validate your JSON objects using the pydantic class definition.
+
+  ```py
+  from docling_core.types import Document
+
+  data_dict = {...}  # here the object you want to validate, as a dictionary
+  Document.model_validate(data_dict)
+
+  data_str = {...}  # here the object as a JSON string
+  Document.model_validate_json(data_str)
+  ```
+
+- You can generate the JSON schema of a model with the script `ds_generate_jsonschema`.
+
+  ```py
+  # for the `Document` type
+  ds_generate_jsonschema Document
+
+  # for the use `Record` type
+  ds_generate_jsonschema Record
+  ```
+
+## Documentation
+
+Docling supports 3 main data types:
+
+- **Document** for publications like books, articles, reports, or patents. When Docling converts an unstructured PDF document, the generated JSON follows this schema.
+  The Document type also models the metadata that may be attached to the converted document.
+  Check [Document](docs/Document.md) for the full JSON schema. 
+- **Record** for structured database records, centered on an entity or _subject_ that is provided with a list of attributes.
+  Related to records, the statements can represent annotations on text by Natural Language Processing (NLP) tools.
+  Check [Record](docs/Record.md) for the full JSON schema. 
+- **Generic** for any data representation, ensuring minimal configuration and maximum flexibility.
+  Check [Generic](docs/Generic.md) for the full JSON schema. 
+
+The data schemas are defined using [pydantic](https://pydantic-docs.helpmanual.io/) models, which provide built-in processes to support the creation of data that adhere to those models.
+
+## Contributing
+
+Please read [Contributing to Docling Core](./CONTRIBUTING.md) for details.
+
+## References
+
+If you use `Docling Core` in your projects, please consider citing the following:
+
+```bib
+@software{Docling,
+author = {Deep Search Team},
+month = {7},
+title = {{Docling}},
+url = {https://github.com/DS4SD/docling},
+version = {main},
+year = {2024}
+}
+```
+
+## License
+
+The `Docling Core` codebase is under MIT license.
+For individual model usage, please refer to the model licenses found in the original packages.

docling_core-0.0.1/docling_core/__init__.py

@@ -0,0 +1,6 @@
+#
+# Copyright IBM Corp. 2024 - 2024
+# SPDX-License-Identifier: MIT
+#
+
+"""Main package."""

docling_core-0.0.1/docling_core/resources/schemas/doc/ANN.json

@@ -0,0 +1,171 @@
+{
+  "$schema": "http://json-schema.org/schema#",
+  "definitions": {
+    "annot+pred": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": [
+          "cells",
+          "clusters",
+          "tables",
+          "source"
+        ],
+        "properties": {
+          "cells": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "required": [
+                "id",
+                "rawcell_id",
+                "label"
+              ],
+              "properties": {
+                "id": {
+                  "type": "integer"
+                },
+                "rawcell_id": {
+                  "type": "integer"
+                },
+                "label": {
+                  "type": "string"
+                }
+              }
+            }
+          },
+          "clusters": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "required": [
+                "model",
+                "type",
+                "bbox",
+                "cell_ids",
+                "merged",
+                "id"
+              ],
+              "properties": {
+                "model": {
+                  "type": "string"
+                },
+                "type": {
+                  "type": "string"
+                },
+                "bbox": {
+                  "type": "array",
+                  "minItems": 4,
+                  "maxItems": 4,
+                  "items": {
+                    "type": "number"
+                  }
+                },
+                "cell_ids": {
+                  "type": "array",
+                  "items": {
+                    "type": "integer"
+                  }
+                },
+                "merged": {
+                  "type": "boolean"
+                },
+                "id": {
+                  "type": "integer"
+                }
+              }
+            }
+          },
+          "tables": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "required": [
+                "cell_id",
+                "label",
+                "rows",
+                "cols"
+              ],
+              "properties": {
+                "cell_id": {
+                  "type": "integer"
+                },
+                "label": {
+                  "type": "string"
+                },
+                "rows": {
+                  "type": "array",
+                  "items": {
+                    "type": "integer"
+                  }
+                },
+                "cols": {
+                  "type": "array",
+                  "items": {
+                    "type": "integer"
+                  }
+                }
+              }
+            }
+          },
+          "source": {
+            "type": "object",
+            "required": [
+              "type",
+              "info",
+              "timestamp"
+            ],
+            "properties": {
+              "type": {
+                "type": "string"
+              },
+              "timestamp": {
+                "type": "number"
+              },
+              "info": {
+                "type": "object",
+                "required": [
+                  "display_name",
+                  "model_name",
+                  "model_class",
+                  "model_version",
+                  "model_id"
+                ],
+                "properties": {
+                  "display_name": {
+                    "type": "string"
+                  },
+                  "model_name": {
+                    "type": "string"
+                  },
+                  "model_class": {
+                    "type": "string"
+                  },
+                  "model_version": {
+                    "type": "string"
+                  },
+                  "model_id": {
+                    "type": "string"
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  },
+  "properties": {
+    "annotations": {
+      "$ref": "#/definitions/annot+pred"
+    },
+    "predictions": {
+      "$ref": "#/definitions/annot+pred"
+    },
+    "reports": {
+      "type": "array"
+    }
+  },
+  "minProperties": 1,
+  "type": "object"
+}