extractor-api-lib 3.3.0__tar.gz → 3.4.0__tar.gz

extractor_api_lib-3.4.0/PKG-INFO

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: extractor-api-lib
+Version: 3.4.0
+Summary: Extracts the content of documents, websites, etc and maps it to a common format.
+License: Apache-2.0
+Author: STACKIT GmbH & Co. KG
+Author-email: data-ai@stackit.cloud
+Maintainer: Andreas Klos
+Maintainer-email: andreas.klos@stackit.cloud
+Requires-Python: >=3.13,<3.14
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: atlassian-python-api (>=4.0.3,<5.0.0)
+Requires-Dist: boto3 (>=1.38.10,<2.0.0)
+Requires-Dist: botocore (>=1.38.10,<2.0.0)
+Requires-Dist: camelot-py[cv] (>=1.0.0,<2.0.0)
+Requires-Dist: datasets (>=3.5.1,<4.0.0)
+Requires-Dist: debugpy (>=1.8.14,<2.0.0)
+Requires-Dist: dependency-injector (>=4.46.0,<5.0.0)
+Requires-Dist: docling (==2.61.2)
+Requires-Dist: docx2txt (>=0.9,<0.10)
+Requires-Dist: fake-useragent (>=2.2.0,<3.0.0)
+Requires-Dist: fastapi (>=0.118.0,<0.119.0)
+Requires-Dist: fasttext (>=0.9.3,<0.10.0)
+Requires-Dist: html5lib (>=1.1,<2.0)
+Requires-Dist: langchain-community (>=0.3.23,<0.4.0)
+Requires-Dist: langchain-core (==0.3.77)
+Requires-Dist: lxml (>=5.4.0,<6.0.0)
+Requires-Dist: markdownify (>=1.1.0,<2.0.0)
+Requires-Dist: markitdown[all] (>=0.1.3,<0.2.0)
+Requires-Dist: numpy (>=2.2.5,<3.0.0)
+Requires-Dist: oauthlib (>=3.2.2,<4.0.0)
+Requires-Dist: opencv-python-headless (==4.12.0.88)
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Requires-Dist: partial (>=1.0,<2.0)
+Requires-Dist: pdf2image (==1.17.0)
+Requires-Dist: pdfplumber (==0.11.7)
+Requires-Dist: pydantic-settings (>=2.9.1,<3.0.0)
+Requires-Dist: pypandoc (>=1.6.2)
+Requires-Dist: pypandoc-binary (>=1.15,<2.0)
+Requires-Dist: pypandoc_binary (>=1.6.2)
+Requires-Dist: pypdfium2 (==4.30.0)
+Requires-Dist: pytesseract (>=0.3.10,<0.4.0)
+Requires-Dist: python-multipart (>=0.0.20,<0.0.21)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: requests-oauthlib (>=2.0.0,<3.0.0)
+Requires-Dist: starlette (>=0.47.2,<0.49.0)
+Requires-Dist: tabulate (>=0.9.0,<0.10.0)
+Requires-Dist: tesserocr (>=2.9.1,<3.0.0)
+Requires-Dist: torch (==2.9.0+cpu)
+Requires-Dist: torchvision (==0.24.0+cpu)
+Requires-Dist: transformers (>=4.53.3,<5.0.0)
+Requires-Dist: unstructured[docx,pptx] (==0.18.15)
+Requires-Dist: uvicorn (>=0.37.0,<0.38.0)
+Requires-Dist: wheel (>=0.45.1,<0.46.0)
+Project-URL: Homepage, https://pypi.org/project/extractor-api-lib
+Project-URL: Repository, https://github.com/stackitcloud/rag-template
+Description-Content-Type: text/markdown
+
+# extractor-api-lib
+
+Content ingestion layer for the STACKIT RAG template. This library exposes a FastAPI extraction service that ingests raw documents (files or remote sources), extracts and converts (to internal representations) the information, and hands output to [`admin-api-lib`](../admin-api-lib/).
+
+## Responsibilities
+
+- Receive binary uploads and remote source descriptors from the admin backend.
+- Route each request through the appropriate extractor (file, sitemap, Confluence, etc.).
+- Convert extracted fragments into the shared `InformationPiece` schema expected by downstream services.
+
+## Feature highlights
+
+- **Layered extraction pipeline** – Docling, MarkItDown, and the custom extractors now cooperate with a deterministic fallback chain, so a failed run automatically cascades to the next extractor.
+- **Expanded format coverage** – PDFs, Office documents, EPUB, XML, Markdown/AsciiDoc, CSV/TXT, raster images, Confluence spaces, and sitemap-driven websites.
+- **Consistent output schema** – Information pieces are returned in a unified structure with content type (`TEXT`, `TABLE`, `IMAGE`) and metadata.
+- **Swappable extractors** – Dependency-injector container makes it easy to add or replace file/source extractors, table converters, etc.
+- **Production-grade plumbing** – Built-in S3-compatible file service, LangChain loaders with retry/backoff, optional PDF OCR, and throttling controls for web crawls.
+
+## File extractor pipeline
+
+[`GeneralFileExtractor`](src/extractor_api_lib/impl/api_endpoints/general_file_extractor.py) orchestrates file parsing. It resolves the file type from the extension, filters the extractors that declare matching `compatible_file_types`, reverses that filtered list, and then executes the extractors in sequence until one returns content or all have failed. Exceptions are logged and the next extractor takes over; only if every extractor either returns no content or raises an exception do we bubble up an error.
+
+### Default execution order
+
+The dependency container wires extractors in the following list:
+
+1. `DoclingFileExtractor`
+2. `MarkitdownFileExtractor`
+3. `PDFExtractor`
+4. `EpubExtractor`
+5. `XMLExtractor`
+6. `MSDocsExtractor`
+7. `TesseractImageExtractor`
+
+Because the orchestrator reverses the candidate list before the fallback loop, the priority for overlapping formats is the reverse of this wiring. For example, PDFs run through Docling first, then fall back to MarkItDown, and finally to the custom PDF extractor; DOCX/PPTX files follow Docling → MarkItDown → MSDocs; raster images go through Docling’s OCR pipeline before falling back to the Tesseract-only extractor.
+
+### Supported formats
+
+| Format family            | Extensions                                               | Primary extractor          | Fallbacks                                                | Notes |
+|--------------------------|----------------------------------------------------------|----------------------------|----------------------------------------------------------|-------|
+| PDF                      | `.pdf`                                                   | Docling                    | MarkItDown → Custom PDF extractor                        | Docling performs OCR + table extraction; the PDF extractor keeps Camelot/pdfplumber heuristics as a last resort. |
+| Microsoft Word           | `.docx`                                                  | Docling                    | MarkItDown → MSDocs                                      | MSDocs keeps unstructured-based table conversion for custom cases. |
+| Microsoft PowerPoint     | `.pptx`                                                  | Docling                    | MarkItDown → MSDocs                                      | MarkItDown splits slides by `<!-- Slide number: N -->`. |
+| Microsoft Excel          | `.xlsx`                                                  | Docling                    | —                                                        | Tables returned as markdown; Docling infers sheet structure. |
+| EPUB                     | `.epub`                                                  | MarkItDown                 | EPUB extractor                                           | MarkItDown covers simple ebooks; the LangChain-based EPUB extractor preserves metadata when MarkItDown fails. |
+| HTML                     | `.html`                                                  | Docling                    | MarkItDown                                               | Docling keeps DOM-aware segmentation; MarkItDown is lighter-weight. |
+| Markdown                 | `.md`, `.markdown`, `.mdx`                               | Docling                    | —                                                        | MarkItDown does not currently register for Markdown. |
+| AsciiDoc                 | `.adoc`, `.asciidoc`                                     | Docling                    | —                                                        | |
+| CSV                      | `.csv`                                                   | Docling                    | MarkItDown                                               | Both produce markdown tables; Docling preserves structured metadata. |
+| Plain text               | `.txt`                                                   | MarkItDown                 | —                                                        | |
+| XML                      | `.xml`                                                   | XML extractor              | —                                                        | Uses the unstructured XML partitioner. |
+| Raster images            | `.jpg`, `.jpeg`, `.png`, `.tiff`, `.tif`, `.bmp`          | Docling (OCR)              | Tesseract image extractor                                | Docling feeds Tesseract CLI OCR; the fallback enforces single-frame images via Pillow. |
+
+Image coverage currently excludes animated GIF, WebP, HEIC, and SVG files. These extensions are ignored by the routing logic and will surface as “No extractor found” errors until an extractor declares support.
+
+### Source extractor pipeline
+
+`GeneralSourceExtractor` wires Confluence and sitemap loaders behind a similar abstraction. Unlike files, source extractors are keyed by `ExtractionParameters.source_type` and the matching extractor is called directly (no fallback chain).
+
+## Configuring extractor order
+
+The order lives in `DependencyContainer.file_extractors`. You can override it either by subclassing the container or by overriding the provider at runtime before wiring the FastAPI app. Example:
+
+`container.py`
+
+```python
+from dependency_injector.providers import List
+
+from extractor_api_lib.dependency_container import DependencyContainer
+
+
+class CustomExtractorContainer(DependencyContainer):
+    file_extractors = List(
+        DependencyContainer.docling_extractor,
+        DependencyContainer.markitdown_extractor,
+        DependencyContainer.ms_docs_extractor,
+        DependencyContainer.pdf_extractor,
+        DependencyContainer.image_extractor,
+        DependencyContainer.xml_extractor,
+        DependencyContainer.epub_extractor,
+    )
+```
+
+`main.py`
+
+```python
+from extractor_api_lib.main import app as perfect_extractor_app, register_dependency_container
+
+from container import CustomExtractorContainer
+
+register_dependency_container(CustomExtractorContainer())
+```
+
+The last provider in the list becomes the first extractor tried for a matching file type. Keep shared singleton providers (file service, converters) in the parent class to avoid double instantiation.
+
+## Installation
+
+```bash
+pip install extractor-api-lib
+```
+
+Python 3.13 is required. OCR and computer-vision features expect system packages such as `ffmpeg`, `poppler-utils`, and `tesseract` (see `services/document-extractor/README.md` for the full list).
+
+## Module tour
+
+- `dependency_container.py` – Central dependency-injector wiring. Override providers here to plug in custom extractors, endpoints etc.
+- `api_endpoints/` & `impl/api_endpoints/` – Thin FastAPI endpoint abstractions and implementations for file and source (like confluence & sitemaps) extractors.
+- `apis/` – Extractor API abstractions and implementations.
+- `extractors/` & `impl/extractors/` – Format-specific logic (PDF, DOCX, PPTX, XML, EPUB, Confluence, sitemap) packaged behind the `InformationExtractor`/`InformationFileExtractor` interfaces.
+- `mapper/` & `impl/mapper/` – Abstractions and implementations to map langchain documents, internal and external information piece representations to each other.
+- `file_services/` – Default S3-compatible storage adapter; replace it if you store files elsewhere.
+- `impl/settings/` – Configuration settings for dependency injection container components.
+- `table_converter/` & `impl/table_converter/` – Abstractions and implementations to convert `pandas.DataFrame` to markdown and vice versa.
+- `impl/types/` - Enums for content-, extractor- and file types.
+- `impl/utils/` – Helper functions for hashed datetime and sitemap crawling, header injection, and custom metadata parsing.
+
+## Endpoints provided
+
+- `POST /extract_from_file` – Downloads the file from S3, extracts its contents, and returns normalized `InformationPiece` records.
+- `POST /extract_from_source` – Pulls from remote sources (Confluence, sitemap) using credentials and further optional kwargs.
+
+Both endpoints stream their results back to `admin-api-lib`, which takes care of enrichment and persistence.
+
+## How the file extraction endpoint works
+
+1. Download the file from S3.
+2. Derive the file type from the extension (normalizing common image/Markdown/AsciiDoc aliases).
+3. Select extractors that declare support for the resolved `FileType`.
+4. Run the extractors in priority order (highest priority first); stop at the first non-empty result or keep falling back if an extractor raises.
+5. Map the internal representation to the external schema and return the final output.
+
+## How the source extraction endpoint works
+
+1. Chose suitable source extractor based on the source type
+2. Pull the source content using the provided credentials and parameters
+3. Extract the content from the source
+4. Map the internal representation to the external schema
+5. Return the final output
+
+## Configuration overview
+
+Two `pydantic-settings` models ship with this package:
+
+- **S3 storage** (`S3Settings`) – configure the built-in file service with `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_ENDPOINT`, and `S3_BUCKET`.
+
+Other extractors accept their parameters at runtime through the request payload (`ExtractionParameters`). For example, the admin backend forwards Confluence credentials, sitemap URLs, or custom headers when it calls `/extract_from_source`. This keeps the library stateless and makes it easy to plug in additional sources without redeploying.
+
+The Helm chart exposes the environment variables mentioned above under `documentExtractor.envs.*` so production deployments remain declarative.
+
+## Typical usage
+
+```python
+from extractor_api_lib.main import app as perfect_extractor_app
+```
+
+`admin-api-lib` calls `/extract_from_file` and `/extract_from_source` to populate the ingestion pipeline.
+
+## Extending the library
+
+1. Implement `InformationFileExtractor` (for file-based inputs) or `InformationExtractor` (for remote sources).
+2. Add a provider to `DependencyContainer` (usually a `Singleton`) and wire dependencies such as the shared `FileService` or table converter.
+3. Append the provider to `file_extractors` (or to the source extractor list) in the desired position so that the fallback order is correct.
+4. Update mappers or metadata handling if additional fields are required.
+5. Cover the happy path and a failure edge case with tests under `libs/extractor-api-lib/tests`, mocking external services (OCR, network, file I/O).
+
+## Advantages and caveats
+
+- Docling-first prioritisation dramatically improves structured extraction (tables, headings) and adds OCR to formats that previously lacked it.
+- Retaining MarkItDown and the custom PDF/MS extractors provides graceful degradation when Docling fails or produces empty output.
+- Image support now goes through Docling’s OCR before falling back to pure Tesseract.
+- The configuration still requires code changes; there is no environment-variable switch to reshuffle or disable extractors at runtime.
+- Multi-frame images, animated/novel image formats, and office formats such as ODT/RTF remain unsupported.
+
+## Contributing
+
+Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
+
+## License
+
+Licensed under the project license. See the root [`LICENSE`](https://github.com/stackitcloud/rag-template/blob/main/LICENSE) file.
+

extractor_api_lib-3.4.0/README.md

@@ -0,0 +1,180 @@
+# extractor-api-lib
+
+Content ingestion layer for the STACKIT RAG template. This library exposes a FastAPI extraction service that ingests raw documents (files or remote sources), extracts and converts (to internal representations) the information, and hands output to [`admin-api-lib`](../admin-api-lib/).
+
+## Responsibilities
+
+- Receive binary uploads and remote source descriptors from the admin backend.
+- Route each request through the appropriate extractor (file, sitemap, Confluence, etc.).
+- Convert extracted fragments into the shared `InformationPiece` schema expected by downstream services.
+
+## Feature highlights
+
+- **Layered extraction pipeline** – Docling, MarkItDown, and the custom extractors now cooperate with a deterministic fallback chain, so a failed run automatically cascades to the next extractor.
+- **Expanded format coverage** – PDFs, Office documents, EPUB, XML, Markdown/AsciiDoc, CSV/TXT, raster images, Confluence spaces, and sitemap-driven websites.
+- **Consistent output schema** – Information pieces are returned in a unified structure with content type (`TEXT`, `TABLE`, `IMAGE`) and metadata.
+- **Swappable extractors** – Dependency-injector container makes it easy to add or replace file/source extractors, table converters, etc.
+- **Production-grade plumbing** – Built-in S3-compatible file service, LangChain loaders with retry/backoff, optional PDF OCR, and throttling controls for web crawls.
+
+## File extractor pipeline
+
+[`GeneralFileExtractor`](src/extractor_api_lib/impl/api_endpoints/general_file_extractor.py) orchestrates file parsing. It resolves the file type from the extension, filters the extractors that declare matching `compatible_file_types`, reverses that filtered list, and then executes the extractors in sequence until one returns content or all have failed. Exceptions are logged and the next extractor takes over; only if every extractor either returns no content or raises an exception do we bubble up an error.
+
+### Default execution order
+
+The dependency container wires extractors in the following list:
+
+1. `DoclingFileExtractor`
+2. `MarkitdownFileExtractor`
+3. `PDFExtractor`
+4. `EpubExtractor`
+5. `XMLExtractor`
+6. `MSDocsExtractor`
+7. `TesseractImageExtractor`
+
+Because the orchestrator reverses the candidate list before the fallback loop, the priority for overlapping formats is the reverse of this wiring. For example, PDFs run through Docling first, then fall back to MarkItDown, and finally to the custom PDF extractor; DOCX/PPTX files follow Docling → MarkItDown → MSDocs; raster images go through Docling’s OCR pipeline before falling back to the Tesseract-only extractor.
+
+### Supported formats
+
+| Format family            | Extensions                                               | Primary extractor          | Fallbacks                                                | Notes |
+|--------------------------|----------------------------------------------------------|----------------------------|----------------------------------------------------------|-------|
+| PDF                      | `.pdf`                                                   | Docling                    | MarkItDown → Custom PDF extractor                        | Docling performs OCR + table extraction; the PDF extractor keeps Camelot/pdfplumber heuristics as a last resort. |
+| Microsoft Word           | `.docx`                                                  | Docling                    | MarkItDown → MSDocs                                      | MSDocs keeps unstructured-based table conversion for custom cases. |
+| Microsoft PowerPoint     | `.pptx`                                                  | Docling                    | MarkItDown → MSDocs                                      | MarkItDown splits slides by `<!-- Slide number: N -->`. |
+| Microsoft Excel          | `.xlsx`                                                  | Docling                    | —                                                        | Tables returned as markdown; Docling infers sheet structure. |
+| EPUB                     | `.epub`                                                  | MarkItDown                 | EPUB extractor                                           | MarkItDown covers simple ebooks; the LangChain-based EPUB extractor preserves metadata when MarkItDown fails. |
+| HTML                     | `.html`                                                  | Docling                    | MarkItDown                                               | Docling keeps DOM-aware segmentation; MarkItDown is lighter-weight. |
+| Markdown                 | `.md`, `.markdown`, `.mdx`                               | Docling                    | —                                                        | MarkItDown does not currently register for Markdown. |
+| AsciiDoc                 | `.adoc`, `.asciidoc`                                     | Docling                    | —                                                        | |
+| CSV                      | `.csv`                                                   | Docling                    | MarkItDown                                               | Both produce markdown tables; Docling preserves structured metadata. |
+| Plain text               | `.txt`                                                   | MarkItDown                 | —                                                        | |
+| XML                      | `.xml`                                                   | XML extractor              | —                                                        | Uses the unstructured XML partitioner. |
+| Raster images            | `.jpg`, `.jpeg`, `.png`, `.tiff`, `.tif`, `.bmp`          | Docling (OCR)              | Tesseract image extractor                                | Docling feeds Tesseract CLI OCR; the fallback enforces single-frame images via Pillow. |
+
+Image coverage currently excludes animated GIF, WebP, HEIC, and SVG files. These extensions are ignored by the routing logic and will surface as “No extractor found” errors until an extractor declares support.
+
+### Source extractor pipeline
+
+`GeneralSourceExtractor` wires Confluence and sitemap loaders behind a similar abstraction. Unlike files, source extractors are keyed by `ExtractionParameters.source_type` and the matching extractor is called directly (no fallback chain).
+
+## Configuring extractor order
+
+The order lives in `DependencyContainer.file_extractors`. You can override it either by subclassing the container or by overriding the provider at runtime before wiring the FastAPI app. Example:
+
+`container.py`
+
+```python
+from dependency_injector.providers import List
+
+from extractor_api_lib.dependency_container import DependencyContainer
+
+
+class CustomExtractorContainer(DependencyContainer):
+    file_extractors = List(
+        DependencyContainer.docling_extractor,
+        DependencyContainer.markitdown_extractor,
+        DependencyContainer.ms_docs_extractor,
+        DependencyContainer.pdf_extractor,
+        DependencyContainer.image_extractor,
+        DependencyContainer.xml_extractor,
+        DependencyContainer.epub_extractor,
+    )
+```
+
+`main.py`
+
+```python
+from extractor_api_lib.main import app as perfect_extractor_app, register_dependency_container
+
+from container import CustomExtractorContainer
+
+register_dependency_container(CustomExtractorContainer())
+```
+
+The last provider in the list becomes the first extractor tried for a matching file type. Keep shared singleton providers (file service, converters) in the parent class to avoid double instantiation.
+
+## Installation
+
+```bash
+pip install extractor-api-lib
+```
+
+Python 3.13 is required. OCR and computer-vision features expect system packages such as `ffmpeg`, `poppler-utils`, and `tesseract` (see `services/document-extractor/README.md` for the full list).
+
+## Module tour
+
+- `dependency_container.py` – Central dependency-injector wiring. Override providers here to plug in custom extractors, endpoints etc.
+- `api_endpoints/` & `impl/api_endpoints/` – Thin FastAPI endpoint abstractions and implementations for file and source (like confluence & sitemaps) extractors.
+- `apis/` – Extractor API abstractions and implementations.
+- `extractors/` & `impl/extractors/` – Format-specific logic (PDF, DOCX, PPTX, XML, EPUB, Confluence, sitemap) packaged behind the `InformationExtractor`/`InformationFileExtractor` interfaces.
+- `mapper/` & `impl/mapper/` – Abstractions and implementations to map langchain documents, internal and external information piece representations to each other.
+- `file_services/` – Default S3-compatible storage adapter; replace it if you store files elsewhere.
+- `impl/settings/` – Configuration settings for dependency injection container components.
+- `table_converter/` & `impl/table_converter/` – Abstractions and implementations to convert `pandas.DataFrame` to markdown and vice versa.
+- `impl/types/` - Enums for content-, extractor- and file types.
+- `impl/utils/` – Helper functions for hashed datetime and sitemap crawling, header injection, and custom metadata parsing.
+
+## Endpoints provided
+
+- `POST /extract_from_file` – Downloads the file from S3, extracts its contents, and returns normalized `InformationPiece` records.
+- `POST /extract_from_source` – Pulls from remote sources (Confluence, sitemap) using credentials and further optional kwargs.
+
+Both endpoints stream their results back to `admin-api-lib`, which takes care of enrichment and persistence.
+
+## How the file extraction endpoint works
+
+1. Download the file from S3.
+2. Derive the file type from the extension (normalizing common image/Markdown/AsciiDoc aliases).
+3. Select extractors that declare support for the resolved `FileType`.
+4. Run the extractors in priority order (highest priority first); stop at the first non-empty result or keep falling back if an extractor raises.
+5. Map the internal representation to the external schema and return the final output.
+
+## How the source extraction endpoint works
+
+1. Chose suitable source extractor based on the source type
+2. Pull the source content using the provided credentials and parameters
+3. Extract the content from the source
+4. Map the internal representation to the external schema
+5. Return the final output
+
+## Configuration overview
+
+Two `pydantic-settings` models ship with this package:
+
+- **S3 storage** (`S3Settings`) – configure the built-in file service with `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_ENDPOINT`, and `S3_BUCKET`.
+
+Other extractors accept their parameters at runtime through the request payload (`ExtractionParameters`). For example, the admin backend forwards Confluence credentials, sitemap URLs, or custom headers when it calls `/extract_from_source`. This keeps the library stateless and makes it easy to plug in additional sources without redeploying.
+
+The Helm chart exposes the environment variables mentioned above under `documentExtractor.envs.*` so production deployments remain declarative.
+
+## Typical usage
+
+```python
+from extractor_api_lib.main import app as perfect_extractor_app
+```
+
+`admin-api-lib` calls `/extract_from_file` and `/extract_from_source` to populate the ingestion pipeline.
+
+## Extending the library
+
+1. Implement `InformationFileExtractor` (for file-based inputs) or `InformationExtractor` (for remote sources).
+2. Add a provider to `DependencyContainer` (usually a `Singleton`) and wire dependencies such as the shared `FileService` or table converter.
+3. Append the provider to `file_extractors` (or to the source extractor list) in the desired position so that the fallback order is correct.
+4. Update mappers or metadata handling if additional fields are required.
+5. Cover the happy path and a failure edge case with tests under `libs/extractor-api-lib/tests`, mocking external services (OCR, network, file I/O).
+
+## Advantages and caveats
+
+- Docling-first prioritisation dramatically improves structured extraction (tables, headings) and adds OCR to formats that previously lacked it.
+- Retaining MarkItDown and the custom PDF/MS extractors provides graceful degradation when Docling fails or produces empty output.
+- Image support now goes through Docling’s OCR before falling back to pure Tesseract.
+- The configuration still requires code changes; there is no environment-variable switch to reshuffle or disable extractors at runtime.
+- Multi-frame images, animated/novel image formats, and office formats such as ODT/RTF remain unsupported.
+
+## Contributing
+
+Ensure new endpoints or adapters remain thin and defer to [`rag-core-lib`](../rag-core-lib/) for shared logic. Run `poetry run pytest` and the configured linters before opening a PR. For further instructions see the [Contributing Guide](https://github.com/stackitcloud/rag-template/blob/main/CONTRIBUTING.md).
+
+## License
+
+Licensed under the project license. See the root [`LICENSE`](https://github.com/stackitcloud/rag-template/blob/main/LICENSE) file.

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "extractor-api-lib"
-version = "v3.3.0"
+version = "v3.4.0"
 description = "Extracts the content of documents, websites, etc and maps it to a common format."
 authors = [
     "STACKIT GmbH & Co. KG <data-ai@stackit.cloud>",
@@ -43,6 +43,7 @@ per-file-ignores = """
   ./src/extractor_api_lib/container.py: CCE002,CCE001,
   ./src/extractor_api_lib/apis/extractor_api_base.py: WOT001,
   ./tests/*: S101,E501,
+  **/__init__.py: D107,
 """
 
 [tool.black]
@@ -76,7 +77,7 @@ known_local_folder = ["extractor_api_lib", "rag_core_lib"]
 max-line-length = 120
 
 [tool.poetry.dependencies]
-python = "^3.13"
+python = ">=3.13,<3.14"
 wheel = "^0.45.1"
 botocore = "^1.38.10"
 fasttext = "^0.9.3"
@@ -91,7 +92,8 @@ python-multipart = "^0.0.20"
 oauthlib = "^3.2.2"
 requests-oauthlib = "^2.0.0"
 pdfplumber = "0.11.7"
-opencv-python = "4.12.0.88"
+tesserocr = "^2.9.1"
+opencv-python-headless = "4.12.0.88"
 pdf2image = "1.17.0"
 datasets = "^3.5.1"
 pandas = "^2.2.2"
@@ -112,6 +114,13 @@ fake-useragent = "^2.2.0"
 pypdfium2 = "4.30.0"
 pypandoc-binary = "^1.15"
 starlette = ">=0.47.2,<0.49.0"
+markitdown = {version = "^0.1.3", extras = ["all"]}
+docling = "2.61.2"
+torch =      { version = "2.9.0+cpu",  source = "pytorch_cpu" }
+torchvision = { version = "0.24.0+cpu", source = "pytorch_cpu" }
+transformers = ">=4.53.3,<5.0.0"
+pypandoc = ">=1.6.2"
+pypandoc_binary = ">=1.6.2"
 
 [tool.poetry.group.dev.dependencies]
 pytest = "^8.3.5"
@@ -138,8 +147,8 @@ flake8-wot = "^0.2.0"
 flake8-function-order = "^0.0.5"
 flake8-tidy-imports = "^4.10.0"
 black = "^25.1.0"
-# flake8-logging-format = "^2024.24.12"
-# flake8-docstrings = "^1.7.0"
+flake8-logging-format = "^2024.24.12"
+flake8-docstrings = "^1.7.0"
 
 [tool.poetry.group.tests.dependencies]
 httpx = "^0.28.1"
@@ -149,4 +158,3 @@ log_cli = true
 log_cli_level = "DEBUG"
 pythonpath = "src"
 testpaths = "src/tests"
-

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/src/extractor_api_lib/api_endpoints/file_extractor.py

@@ -1,3 +1,5 @@
+"""Module for the abstract file extractor."""
+
 from abc import ABC, abstractmethod
 from extractor_api_lib.models.extraction_request import ExtractionRequest
 from extractor_api_lib.models.information_piece import InformationPiece

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/src/extractor_api_lib/api_endpoints/source_extractor.py

@@ -1,3 +1,5 @@
+"""Module for the abstract source extractor."""
+
 from abc import ABC, abstractmethod
 
 from extractor_api_lib.models.extraction_parameters import ExtractionParameters

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/src/extractor_api_lib/apis/extractor_api_base.py

@@ -22,6 +22,7 @@ class BaseExtractorApi:
     subclasses: ClassVar[Tuple] = ()
 
     def __init_subclass__(cls, **kwargs):
+        """Register the subclass."""
         super().__init_subclass__(**kwargs)
         BaseExtractorApi.subclasses = BaseExtractorApi.subclasses + (cls,)
 

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/src/extractor_api_lib/dependency_container.py

@@ -13,6 +13,15 @@ from extractor_api_lib.impl.extractors.confluence_extractor import ConfluenceExt
 from extractor_api_lib.impl.extractors.file_extractors.epub_extractor import (
     EpubExtractor,
 )
+from extractor_api_lib.impl.extractors.file_extractors.docling_extractor import (
+    DoclingFileExtractor,
+)
+from extractor_api_lib.impl.extractors.file_extractors.image_extractor import (
+    TesseractImageExtractor,
+)
+from extractor_api_lib.impl.extractors.file_extractors.markitdown_extractor import (
+    MarkitdownFileExtractor,
+)
 from extractor_api_lib.impl.extractors.file_extractors.ms_docs_extractor import (
     MSDocsExtractor,
 )
@@ -32,7 +41,6 @@ from extractor_api_lib.impl.mapper.langchain_document2information_piece import (
 from extractor_api_lib.impl.mapper.sitemap_document2information_piece import (
     SitemapLangchainDocument2InformationPiece,
 )
-from extractor_api_lib.impl.settings.pdf_extractor_settings import PDFExtractorSettings
 from extractor_api_lib.impl.settings.s3_settings import S3Settings
 from extractor_api_lib.impl.table_converter.dataframe2markdown import DataFrame2Markdown
 from extractor_api_lib.impl.utils.sitemap_extractor_utils import (
@@ -46,16 +54,18 @@ class DependencyContainer(DeclarativeContainer):
 
     # Settings
     settings_s3 = S3Settings()
-    settings_pdf_extractor = PDFExtractorSettings()
 
     sitemap_parsing_function = Factory(lambda: custom_sitemap_parser_function)
     sitemap_meta_function = Factory(lambda: custom_sitemap_metadata_parser_function)
 
     database_converter = Singleton(DataFrame2Markdown)
     file_service = Singleton(S3Service, settings_s3)
-    pdf_extractor = Singleton(PDFExtractor, file_service, settings_pdf_extractor, database_converter)
+    pdf_extractor = Singleton(PDFExtractor, file_service, database_converter)
     ms_docs_extractor = Singleton(MSDocsExtractor, file_service, database_converter)
     xml_extractor = Singleton(XMLExtractor, file_service)
+    markitdown_extractor = Singleton(MarkitdownFileExtractor, file_service)
+    docling_extractor = Singleton(DoclingFileExtractor, file_service)
+    image_extractor = Singleton(TesseractImageExtractor, file_service)
 
     intern2external = Singleton(Internal2ExternalInformationPiece)
     confluence_document2information_piece = Singleton(ConfluenceLangchainDocument2InformationPiece)
@@ -63,9 +73,22 @@ class DependencyContainer(DeclarativeContainer):
     sitemap_document2information_piece = Singleton(SitemapLangchainDocument2InformationPiece)
     epub_extractor = Singleton(EpubExtractor, file_service, langchain_document2information_piece)
 
-    file_extractors = List(pdf_extractor, ms_docs_extractor, xml_extractor, epub_extractor)
+    file_extractors = List(
+        image_extractor,
+        ms_docs_extractor,
+        xml_extractor,
+        epub_extractor,
+        pdf_extractor,
+        markitdown_extractor,
+        docling_extractor,
+    )
 
-    general_file_extractor = Singleton(GeneralFileExtractor, file_service, file_extractors, intern2external)
+    general_file_extractor = Singleton(
+        GeneralFileExtractor,
+        file_service=file_service,
+        available_extractors=file_extractors,
+        mapper=intern2external,
+    )
     confluence_extractor = Singleton(ConfluenceExtractor, mapper=confluence_document2information_piece)
 
     sitemap_extractor = Singleton(

{extractor_api_lib-3.3.0 → extractor_api_lib-3.4.0}/src/extractor_api_lib/extractors/information_extractor.py

@@ -13,7 +13,8 @@ class InformationExtractor(ABC):
 
     @property
     @abstractmethod
-    def extractor_type(self) -> ExtractorTypes: ...
+    def extractor_type(self) -> ExtractorTypes:
+        """The type of the extractor."""
 
     @abstractmethod
     async def aextract_content(