extralit-server 0.7.0__tar.gz → 0.7.2__tar.gz

{extralit_server-0.7.0 → extralit_server-0.7.2}/CHANGELOG.md

@@ -18,6 +18,26 @@ These are the section headers that we use:
 ### Added
 
 - Refactor document analysis and preprocessing job to support asynchronous s3 IO operations and large file processing.
+- Adopted `DoclingDocument` as the canonical internal layout model, so every extracted item carries a `page_no` + `bbox` + `charspan` provenance triple that an extraction record can cite.
+- Added swappable layout parsers behind `contexts/ocr/parsers`: `pdf_inspector` (MIT, required, the default) and `pymupdf` (AGPL, optional `pymupdf` extra, the only one yielding per-cell table geometry).
+- Added `async_document_layout_job` on the OCR queue, persisting layout as canonical JSON plus a columnar projection in the workspace's `layout/items.lance` and `layout/pages.lance` datasets, queryable through `LayoutStore` and its DuckDB views as soon as the job commits.
+- Added `GET /documents/{document_id}/layout`, with `pages` and `labels` filters, plus the matching frontend `DocumentLayout` entities and `DocumentRepository.getDocumentLayout`.
+- Added `layout_parser` to `POST /workflows/start` to trigger layout extraction as part of the document workflow.
+
+### Changed
+
+- The analysis job now triages PDFs with pdf-inspector (`pdf_type`, `pages_needing_ocr`, tables, columns) instead of the pdfminer OCR-layer detector, estimates margins over the leading five pages instead of rendering every page, and runs ocrmypdf for page rotation only, with tesseract OCR disabled and a bounded OSD budget. A failed rotation is recorded and the job still succeeds.
+- Layout extraction is enqueued on upload by default; `layout_parser` remains an override.
+- Document workflow dependents (text extraction, layout) now wait on the analysis/preprocessing job with `allow_failure`, and carry their retry policy and a 24 h result TTL through `Queue.prepare_data()` — the `@job` decorator values never applied on that path.
+- `POST /workflows/start?force=true` stops the previous run's jobs before enqueueing new ones, whose ids now carry the run suffix.
+- Deleting a document removes its PDF, thumbnail, layout JSON and layout rows together, through `files.delete_document_artifacts`.
+- Rewrote `contexts/ocr/{text,tables,figures}` as single-item appenders driven by one ordered pass, fixing text inside tables and figures being emitted twice and restoring geometric reading order.
+
+### Removed
+
+- Removed `contexts/document/analysis.py` (`PDFOCRLayerDetector`) and the OCR-only preprocessing knobs (`language`, `force_ocr`, `redo_ocr`, `skip_big`, `pdf_renderer`, `output_type`, `fast_web_view`, `deskew`, `enable_analysis`); `has_ocr_text_layer`, `needs_ocr` and `ocr_quality` are no longer written.
+- Removed the `marker` extra (`marker-pdf`, `torch`, `torchvision`, `transformers`) and the unused `async_marker_layout_job`.
+- Removed the deprecated Unstructured-shaped `document/chunks.py` schemas (`Segments`, `TextSegment`, `TableSegment`, `FigureSegment`, `Coordinates`) and the unreferenced `GPU_QUEUE`.
 
 ### Fixed
 

{extralit_server-0.7.0 → extralit_server-0.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: extralit-server
-Version: 0.7.0
+Version: 0.7.2
 Summary: Open-source tool for accurate & fast scientific literature data extraction with LLM and human-in-the-loop.
 Project-URL: homepage, https://extralit.ai
 Project-URL: documentation, https://docs.extralit.ai
@@ -12,58 +12,58 @@ License-File: LICENSE
 License-File: LICENSE_HEADER
 Keywords: artificial-intelligence,data-annotation,human-in-the-loop,literature-review,machine-learning,mlops
 Requires-Python: >=3.10
-Requires-Dist: aioboto3>=13.1.1
-Requires-Dist: aiofiles~=24.1.0
 Requires-Dist: aiosqlite==0.20.0
-Requires-Dist: alembic~=1.13.0
+Requires-Dist: alembic>=1.13.0
 Requires-Dist: authlib>=1.6.9
-Requires-Dist: bcrypt~=4.2.0
-Requires-Dist: brotli-asgi~=1.4.0
+Requires-Dist: bcrypt>=4.2.0
+Requires-Dist: brotli-asgi>=1.4.0
 Requires-Dist: click>=8.2.0
 Requires-Dist: datasets>=3.6.0
+Requires-Dist: docling-core<3.0.0,>=2.91.0
 Requires-Dist: duckdb>=1.5.4
-Requires-Dist: elasticsearch8[async]~=8.7.0
-Requires-Dist: fastapi~=0.115.0
+Requires-Dist: elasticsearch8[async]>=8.7.0
+Requires-Dist: fastapi>=0.115.0
 Requires-Dist: filelock>=3.25.2
-Requires-Dist: greenlet~=3.1.0
-Requires-Dist: httpx~=0.27.0
-Requires-Dist: huggingface-hub~=0.34.0
+Requires-Dist: greenlet>=3.1.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: huggingface-hub>=0.34.0
 Requires-Dist: jinja2>=3.1.4
-Requires-Dist: lancedb>=0.34.0
+Requires-Dist: lancedb>=0.37.1
 Requires-Dist: lazy-loader>=0.4
 Requires-Dist: litellm<=1.82.6,>=1.80.0
 Requires-Dist: numpy<3.0.0,>=2.0.0
-Requires-Dist: oauthlib~=3.2.0
+Requires-Dist: oauthlib>=3.2.0
+Requires-Dist: obstore>=0.11.0
 Requires-Dist: ocrmypdf>=16.11.0
 Requires-Dist: opencv-python-headless>=4.11.0.86
-Requires-Dist: opensearch-py~=2.0.0
+Requires-Dist: opensearch-py>=2.0.0
 Requires-Dist: packaging>=23.2
-Requires-Dist: pandera[io]>=0.20
+Requires-Dist: pandera[io]>=0.32.0
+Requires-Dist: pdf-inspector>=1.14.2
 Requires-Dist: pdf2image>=1.17.0
 Requires-Dist: pillow>=10.1.0
-Requires-Dist: psutil<5.10,~=5.8
-Requires-Dist: pydantic-settings~=2.6.0
-Requires-Dist: pydantic~=2.9.0
-Requires-Dist: python-jose[cryptography]~=3.3.0
-Requires-Dist: python-multipart~=0.0.16
+Requires-Dist: psutil<5.10,>=5.8
+Requires-Dist: pyarrow>=23.0.1
+Requires-Dist: pydantic-settings>=2.6.0
+Requires-Dist: pydantic>=2.9.0
+Requires-Dist: pylance>=10.0.0
+Requires-Dist: python-jose[cryptography]>=3.3.0
+Requires-Dist: python-multipart>=0.0.16
 Requires-Dist: pyyaml<6.1.0,>=5.4.1
 Requires-Dist: rich!=13.1.0
-Requires-Dist: rq~=2.4.1
-Requires-Dist: social-auth-core~=4.5.0
-Requires-Dist: sqlalchemy~=2.0.0
+Requires-Dist: rq>=2.4.1
+Requires-Dist: social-auth-core>=4.5.0
+Requires-Dist: sqlalchemy>=2.0.0
 Requires-Dist: standardwebhooks>=1.0.0
 Requires-Dist: tenacity>=9.1.2
 Requires-Dist: typer>=0.19.1
-Requires-Dist: types-aiobotocore-s3==2.24.2
-Requires-Dist: uvicorn[standard]~=0.32.0
-Provides-Extra: marker
-Requires-Dist: marker-pdf>=1.9.3; extra == 'marker'
-Requires-Dist: torch>=2.5.0; extra == 'marker'
-Requires-Dist: torchvision>=0.20.0; extra == 'marker'
-Requires-Dist: transformers>=4.51.0; extra == 'marker'
+Requires-Dist: uvicorn[standard]>=0.32.0
+Requires-Dist: xxhash>=3.6.0
 Provides-Extra: postgresql
-Requires-Dist: asyncpg~=0.30.0; extra == 'postgresql'
-Requires-Dist: psycopg2~=2.9.0; extra == 'postgresql'
+Requires-Dist: asyncpg>=0.30.0; extra == 'postgresql'
+Requires-Dist: psycopg2>=2.9.0; extra == 'postgresql'
+Provides-Extra: pymupdf
+Requires-Dist: pymupdf4llm~=0.3.4; extra == 'pymupdf'
 Description-Content-Type: text/markdown
 
 <h1 align="center">