deepdoctection 0.46.2__tar.gz → 1.0.0__tar.gz

deepdoctection-1.0.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: deepdoctection
+Version: 1.0.0
+Summary: Repository for Document AI - server/inference core package
+Author: Dr. Janis Meyer
+License: Apache License 2.0
+Project-URL: Homepage, https://github.com/deepdoctection/deepdoctection
+Project-URL: Documentation, https://deepdoctection.readthedocs.io
+Project-URL: Repository, https://github.com/deepdoctection/deepdoctection
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Natural Language :: English
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: dd-core[full]>=1.0
+Requires-Dist: huggingface_hub>=0.26.0
+Provides-Extra: full
+Requires-Dist: dd-datasets[full]>=1.0; extra == "full"
+Requires-Dist: boto3==1.34.102; extra == "full"
+Requires-Dist: pdfplumber>=0.11.0; extra == "full"
+Requires-Dist: jdeskew>=0.2.2; extra == "full"
+Requires-Dist: networkx>=2.7.1; extra == "full"
+Requires-Dist: apted==1.0.3; extra == "full"
+Requires-Dist: distance==0.1.3; extra == "full"
+Requires-Dist: lxml>=4.9.1; extra == "full"
+Requires-Dist: pycocotools>=2.0.2; extra == "full"
+Requires-Dist: timm>=0.9.16; extra == "full"
+Requires-Dist: transformers<5.0.0,>=4.48.0; extra == "full"
+Requires-Dist: accelerate>=0.29.1; extra == "full"
+Requires-Dist: python-doctr>=1.0.0; extra == "full"
+Provides-Extra: types
+Requires-Dist: dd_core[types]; extra == "types"
+Requires-Dist: lxml-stubs>=0.5.1; extra == "types"
+Provides-Extra: dev
+Requires-Dist: black==25.11.0; extra == "dev"
+Requires-Dist: isort==7.0.0; extra == "dev"
+Requires-Dist: pylint==4.0.2; extra == "dev"
+Requires-Dist: mypy==1.4.1; extra == "dev"
+Requires-Dist: types-PyYAML>=6.0.12.12; extra == "dev"
+Requires-Dist: types-termcolor>=1.1.3; extra == "dev"
+Requires-Dist: types-tabulate>=0.9.0.3; extra == "dev"
+Requires-Dist: types-tqdm>=4.66.0.5; extra == "dev"
+Requires-Dist: types-Pillow>=10.2.0.20240406; extra == "dev"
+Requires-Dist: types-urllib3>=1.26.25.14; extra == "dev"
+Requires-Dist: lxml-stubs>=0.5.1; extra == "dev"
+Provides-Extra: test
+Requires-Dist: pytest==9.0.1; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Provides-Extra: docs
+Requires-Dist: mkdocs-material==9.7.0; extra == "docs"
+Requires-Dist: mkdocstrings-python==1.19.0; extra == "docs"
+Requires-Dist: griffe==0.25.0; extra == "docs"
+
+<p align="center">
+  <img src="https://github.com/deepdoctection/deepdoctection/raw/master/docs/_imgs/dd_logo.png" alt="Deep Doctection Logo" width="60%">
+</p>
+
+
+# deepdoctection
+
+**deepdoctection** is the main package for running and training models. It provides the
+pipeline framework, model wrappers, built-in pipelines, training scripts and evaluation methods.
+
+The base package only installs the necessary dependencies for running inference with some selected models. 
+For training, evaluating as well as running all available models, the full package needs to be installed. 
+
+## Overview
+
+- **analyzer**: Configuration and factory functions for creating document analysis pipelines and the built-in analyzer.
+- **configs**: YAML configuration for pipelines and model profiles for the model catalogue.
+- **extern**: External model wrappers (Detectron2, DocTr, HuggingFace Transformers, Tesseract, PdfPlumber, etc.)
+- **pipe**: Pipeline components and services.
+- **eval**: Evaluation metrics and Evaluator.
+- **train**: Training utilities and training scripts for Detectron2 and selected Transformer models.
+
+
+## Installation
+
+### Basic Installation
+
+For inference use cases, install the base package:
+
+```bash
+uv pip install deepdoctection
+```
+
+**Important**: Various dependencies must be installed separately:
+
+- **PyTorch**: Follow instructions at https://pytorch.org/get-started/locally/ according to your os and hardware.
+- **Transformers**: `pip install transformers>=4.48.0` (if using HF models)
+- **Timm**: `pip install timm>=0.9.16` (necessary for if using some dedicated HF models)
+- **DocTr**: `pip install python-doctr>=1.0.0` (if using DocTr models)
+- **Detectron2**: Follow instructions at https://detectron2.readthedocs.io/en/latest/tutorials/install.html
+- **PDFPlumber**: `pip install pdfplumber>=0.11.0`
+- **JDeskew**: `pip install jdeskew>=0.2.2`
+- **Boto3**: `pip install boto3==1.34.102`
+
+For running evaluation with various metrics you can also install in then use:
+
+- **APTED**: `pip install apted==1.0.3`
+- **Distance**: `pip install distance==0.1.3`
+- **Pycocotools**: `pip install pycocotools>=2.0.2`
+
+Image processing is supported by PIL or OpenCV. PIL is used by default and will always be installed. If 
+you prefer to use OpenCV, you can install it:
+
+- **OpenCV**: `pip install opencv-python==4.8.0.76`
+
+
+### Full Installation (Training & Evaluation)
+
+For a one large install with all dependencies (except PyTorch), run:
+
+```bash
+uv pip install deepdoctection[full]
+```
+
+### Development Installation
+
+For development purpose use clone the repository and install in editable mode.
+
+## License
+
+Apache License 2.0
+
+## Author
+
+Dr. Janis Meyer

deepdoctection-1.0.0/README.md

@@ -0,0 +1,75 @@
+<p align="center">
+  <img src="https://github.com/deepdoctection/deepdoctection/raw/master/docs/_imgs/dd_logo.png" alt="Deep Doctection Logo" width="60%">
+</p>
+
+
+# deepdoctection
+
+**deepdoctection** is the main package for running and training models. It provides the
+pipeline framework, model wrappers, built-in pipelines, training scripts and evaluation methods.
+
+The base package only installs the necessary dependencies for running inference with some selected models. 
+For training, evaluating as well as running all available models, the full package needs to be installed. 
+
+## Overview
+
+- **analyzer**: Configuration and factory functions for creating document analysis pipelines and the built-in analyzer.
+- **configs**: YAML configuration for pipelines and model profiles for the model catalogue.
+- **extern**: External model wrappers (Detectron2, DocTr, HuggingFace Transformers, Tesseract, PdfPlumber, etc.)
+- **pipe**: Pipeline components and services.
+- **eval**: Evaluation metrics and Evaluator.
+- **train**: Training utilities and training scripts for Detectron2 and selected Transformer models.
+
+
+## Installation
+
+### Basic Installation
+
+For inference use cases, install the base package:
+
+```bash
+uv pip install deepdoctection
+```
+
+**Important**: Various dependencies must be installed separately:
+
+- **PyTorch**: Follow instructions at https://pytorch.org/get-started/locally/ according to your os and hardware.
+- **Transformers**: `pip install transformers>=4.48.0` (if using HF models)
+- **Timm**: `pip install timm>=0.9.16` (necessary for if using some dedicated HF models)
+- **DocTr**: `pip install python-doctr>=1.0.0` (if using DocTr models)
+- **Detectron2**: Follow instructions at https://detectron2.readthedocs.io/en/latest/tutorials/install.html
+- **PDFPlumber**: `pip install pdfplumber>=0.11.0`
+- **JDeskew**: `pip install jdeskew>=0.2.2`
+- **Boto3**: `pip install boto3==1.34.102`
+
+For running evaluation with various metrics you can also install in then use:
+
+- **APTED**: `pip install apted==1.0.3`
+- **Distance**: `pip install distance==0.1.3`
+- **Pycocotools**: `pip install pycocotools>=2.0.2`
+
+Image processing is supported by PIL or OpenCV. PIL is used by default and will always be installed. If 
+you prefer to use OpenCV, you can install it:
+
+- **OpenCV**: `pip install opencv-python==4.8.0.76`
+
+
+### Full Installation (Training & Evaluation)
+
+For a one large install with all dependencies (except PyTorch), run:
+
+```bash
+uv pip install deepdoctection[full]
+```
+
+### Development Installation
+
+For development purpose use clone the repository and install in editable mode.
+
+## License
+
+Apache License 2.0
+
+## Author
+
+Dr. Janis Meyer

deepdoctection-1.0.0/pyproject.toml

@@ -0,0 +1,108 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "deepdoctection"
+version = "1.0.0"
+authors = [
+    {name = "Dr. Janis Meyer"}
+]
+description = "Repository for Document AI - server/inference core package"
+readme = "README.md"
+license = {text = "Apache License 2.0"}
+requires-python = ">=3.10"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "License :: OSI Approved :: Apache Software License",
+    "Natural Language :: English",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+dependencies = [
+    "dd-core[full]>=1.0",
+    "huggingface_hub>=0.26.0",
+]
+
+[project.optional-dependencies]
+full = [
+    "dd-datasets[full]>=1.0",
+    # Additional dependencies/ Non DL predictors
+    "boto3==1.34.102",
+    "pdfplumber>=0.11.0",
+    "jdeskew>=0.2.2",
+    "networkx>=2.7.1",
+    # Some libs for evals
+    "apted==1.0.3",
+    "distance==0.1.3",
+    "lxml>=4.9.1",
+    "pycocotools>=2.0.2",
+    # DL dependencies
+    "timm>=0.9.16",
+    "transformers>=4.48.0,<5.0.0",
+    "accelerate>=0.29.1",
+    "python-doctr>=1.0.0",
+]
+
+types = [
+    "dd_core[types]",
+    "lxml-stubs>=0.5.1",
+]
+
+dev = [
+    "black==25.11.0",
+    "isort==7.0.0",
+    "pylint==4.0.2",
+    "mypy==1.4.1",
+    "types-PyYAML>=6.0.12.12",
+    "types-termcolor>=1.1.3",
+    "types-tabulate>=0.9.0.3",
+    "types-tqdm>=4.66.0.5",
+    "types-Pillow>=10.2.0.20240406",
+    "types-urllib3>=1.26.25.14",
+    "lxml-stubs>=0.5.1",
+]
+
+test = [
+    "pytest==9.0.1",
+    "pytest-cov",
+]
+
+docs = [
+    "mkdocs-material==9.7.0",
+    "mkdocstrings-python==1.19.0",
+    "griffe==0.25.0"
+]
+
+[project.urls]
+Homepage = "https://github.com/deepdoctection/deepdoctection"
+Documentation = "https://deepdoctection.readthedocs.io"
+Repository = "https://github.com/deepdoctection/deepdoctection"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+deepdoctection = ["py.typed", "configs/*.yaml", "configs/*.jsonl"]
+
+[tool.black]
+line-length = 120
+target-version = ['py310']
+
+[tool.isort]
+profile = "black"
+line_length = 120
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true
+ignore_missing_imports = true
+

deepdoctection-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

deepdoctection-1.0.0/src/deepdoctection/__init__.py

@@ -0,0 +1,176 @@
+# -*- coding: utf-8 -*-
+# File: __init__.py
+
+"""
+Init file for deepdoctection package. This file is used to import all submodules and to set some environment variables
+"""
+
+import sys
+from typing import TYPE_CHECKING, Dict, List
+
+from dd_core.utils.env_info import collect_env_info
+from dd_core.utils.file_utils import _LazyModule
+from dd_core.utils.logger import LoggingRecord, logger
+
+__version__ = "1.0.0"
+_IMPORT_STRUCTURE = {
+    "analyzer": ["config_sanity_checks", "get_dd_analyzer", "ServiceFactory", "update_cfg_from_defaults"],
+    "eval": [
+        "AccuracyMetric",
+        "ConfusionMetric",
+        "PrecisionMetric",
+        "RecallMetric",
+        "F1Metric",
+        "PrecisionMetricMicro",
+        "RecallMetricMicro",
+        "F1MetricMicro",
+        "MetricBase",
+        "CocoMetric",
+        "Evaluator",
+        "metric_registry",
+        "get_metric",
+        "TableTree",
+        "CustomConfig",
+        "TEDS",
+        "TedsMetric",
+    ],
+    "extern": [
+        "ModelCategories",
+        "NerModelCategories",
+        "PredictorBase",
+        "DetectionResult",
+        "ObjectDetector",
+        "PdfMiner",
+        "TextRecognizer",
+        "TokenClassResult",
+        "SequenceClassResult",
+        "LMTokenClassifier",
+        "LMSequenceClassifier",
+        "LanguageDetector",
+        "ImageTransformer",
+        "DeterministicImageTransformer",
+        "InferenceResize",
+        "D2FrcnnDetector",
+        "D2FrcnnTracingDetector",
+        "Jdeskewer",
+        "DoctrTextlineDetector",
+        "DoctrTextRecognizer",
+        "DocTrRotationTransformer",
+        "HFDetrDerivedDetector",
+        "get_tokenizer_from_architecture",
+        "HFLayoutLmTokenClassifierBase",
+        "HFLayoutLmTokenClassifier",
+        "HFLayoutLmv2TokenClassifier",
+        "HFLayoutLmv3TokenClassifier",
+        "HFLayoutLmSequenceClassifier",
+        "HFLayoutLmv2SequenceClassifier",
+        "HFLayoutLmv3SequenceClassifier",
+        "HFLiltTokenClassifier",
+        "HFLiltSequenceClassifier",
+        "HFLmTokenClassifier",
+        "HFLmSequenceClassifier",
+        "HFLmLanguageDetector",
+        "ModelProfile",
+        "ModelCatalog",
+        "print_model_infos",
+        "ModelDownloadManager",
+        "PdfPlumberTextDetector",
+        "Pdfmium2TextDetector",
+        "TesseractOcrDetector",
+        "TesseractRotationTransformer",
+        "TextractOcrDetector",
+    ],
+    "pipe": [
+        "DatapointManager",
+        "PipelineComponent",
+        "PredictorPipelineComponent",
+        "LanguageModelPipelineComponent",
+        "ImageTransformPipelineComponent",
+        "Pipeline",
+        "DetectResultGenerator",
+        "SubImageLayoutService",
+        "ImageCroppingService",
+        "IntersectionMatcher",
+        "NeighbourMatcher",
+        "FamilyCompound",
+        "MatchingService",
+        "PageParsingService",
+        "AnnotationNmsService",
+        "MultiThreadPipelineComponent",
+        "DoctectionPipe",
+        "LanguageDetectionService",
+        "skip_if_category_or_service_extracted",
+        "ImageLayoutService",
+        "LMTokenClassifierService",
+        "LMSequenceClassifierService",
+        "OrderGenerator",
+        "TextLineGenerator",
+        "TextLineService",
+        "TextOrderService",
+        "TableSegmentationRefinementService",
+        "generate_html_string",
+        "pipeline_component_registry",
+        "TableSegmentationService",
+        "PubtablesSegmentationService",
+        "SegmentationResult",
+        "TextExtractionService",
+        "SimpleTransformService",
+    ],
+    "train": [
+        "D2Trainer",
+        "train_d2_faster_rcnn",
+        "LayoutLMTrainer",
+        "train_hf_layoutlm",
+        "DetrDerivedTrainer",
+        "train_hf_detr",
+    ],
+}
+
+
+# Setting some environment variables so that standard functions can be invoked with available hardware
+env_info = collect_env_info()
+logger.debug(LoggingRecord(msg=env_info))
+
+# Build extra objects for the lazy module, starting with the version
+_extra_objects: Dict[str, object] = {"__version__": __version__}
+
+# Re-export all public attributes from dd_core under deepdoctection namespace
+import dd_core  # pylint: disable=C0413
+
+for _name in dir(dd_core):
+    if _name.startswith("_"):
+        continue
+    # Optional: if dd_core defines __all__, you could respect it instead:
+    # if hasattr(dd_core, "__all__") and _name not in dd_core.__all__:
+    #     continue
+    _extra_objects[_name] = getattr(dd_core, _name)
+
+try:
+    import dd_datasets
+
+    for _name in dir(dd_datasets):
+        if _name.startswith("_"):
+            continue
+        _extra_objects[_name] = getattr(dd_datasets, _name)
+except ImportError:
+    pass
+
+# Direct imports for type-checking
+if TYPE_CHECKING:
+    from dd_core import *
+    from dd_datasets import *
+
+    from .analyzer import *
+    from .eval import *
+    from .extern import *  # type: ignore
+    from .pipe import *
+    from .train import *
+
+else:
+    sys.modules[__name__] = _LazyModule(
+        __name__,
+        globals()["__file__"],
+        _IMPORT_STRUCTURE,
+        module_spec=globals().get("__spec__"),
+        extra_objects=_extra_objects,
+    )