spartan-tables 2.0.0__tar.gz

spartan_tables-2.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shlok Nandurbarkar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spartan_tables-2.0.0/PKG-INFO

@@ -0,0 +1,396 @@
+Metadata-Version: 2.4
+Name: spartan-tables
+Version: 2.0.0
+Summary: SPARTAN: heuristic, training-free table detection and extraction for PDF documents
+Author: Shlok Nandurbarkar
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/shlok-n/spartan
+Project-URL: Issues, https://github.com/shlok-n/spartan/issues
+Project-URL: Changelog, https://github.com/shlok-n/spartan/blob/main/CHANGELOG.md
+Project-URL: Paper, https://doi.org/10.1038/s41598-026-44325-7
+Keywords: table detection,table extraction,pdf,opencv,ocr,document ai
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Classifier: Topic :: Text Processing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: opencv-python>=4.8
+Requires-Dist: pypdfium2>=4.25
+Requires-Dist: pydantic>=2.5
+Requires-Dist: pydantic-settings>=2.1
+Provides-Extra: ocr
+Requires-Dist: pytesseract>=0.3.10; extra == "ocr"
+Provides-Extra: api
+Requires-Dist: fastapi>=0.110; extra == "api"
+Requires-Dist: uvicorn>=0.27; extra == "api"
+Requires-Dist: python-multipart>=0.0.9; extra == "api"
+Provides-Extra: llm
+Requires-Dist: ollama>=0.3; extra == "llm"
+Provides-Extra: azure
+Requires-Dist: azure-cognitiveservices-vision-computervision>=0.9; extra == "azure"
+Requires-Dist: msrest>=0.7; extra == "azure"
+Requires-Dist: openai>=1.30; extra == "azure"
+Provides-Extra: metrics
+Requires-Dist: psutil>=5.9; extra == "metrics"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: httpx>=0.27; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5; extra == "dev"
+Provides-Extra: all
+Requires-Dist: spartan-tables[api,llm,metrics,ocr]; extra == "all"
+Dynamic: license-file
+
+# SPARTAN — Structured Parsing and Relevant Table Analysis
+
+[![tests](https://github.com/shlok-n/spartan/actions/workflows/tests.yml/badge.svg)](https://github.com/shlok-n/spartan/actions/workflows/tests.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/shlok-n/spartan/blob/main/LICENSE)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)
+[![Paper](https://img.shields.io/badge/paper-Scientific%20Reports-informational.svg)](https://doi.org/10.1038/s41598-026-44325-7)
+[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](https://github.com/shlok-n/spartan/blob/main/CONTRIBUTING.md)
+
+Heuristic, training-free table detection and extraction for PDF documents. No GPU,
+no model training, fully explainable.
+
+> **Open source and open to contributions.** Bug reports, difficult sample documents, new
+> OCR engines and better heuristics are all welcome — see [Contributing](#contributing).
+
+Reference implementation of:
+
+> Nandurbarkar, S., Chaudhari, A. Y. & Mulla, R. *SPARTAN: automated table detection and
+> extraction from documents using advanced OpenCV heuristics and OCR techniques.*
+> Scientific Reports **16**, 14954 (2026). https://doi.org/10.1038/s41598-026-44325-7
+
+---
+
+## Highlights
+
+- **Exact text for born-digital PDFs.** Characters come from the PDF text layer and are
+  mapped onto the detected cell grid, so part numbers are reproduced exactly (no
+  `0`/`O` or `1`/`I` OCR confusions). OCR (Tesseract, offline) is used only for scanned
+  pages or for regions that have no text layer.
+- **Bordered, borderless and fill-delimited tables**: morphological boundary detector,
+  alignment-based borderless detector with pseudo boundaries, and adaptive thresholding
+  for tables whose cells are delimited only by background colours.
+- **Complex structure**: nested (multi-level) headers, row/column-spanning cells, title
+  rows, section rows, key-value tables and form grids, header-less list tables, and tables
+  drawn as several separate boxes.
+- **Document-level post-processing**: tables continuing across pages are merged (repeated
+  header rows removed), and repeated page headers/footers are detected and de-duplicated.
+- **Optional PCN schema mapping** (the paper's "LLM hook"). Deterministic keyword
+  matching first; a local LLM through Ollama (or Azure OpenAI) fills the remaining gaps.
+- CLI, Python API and REST API (FastAPI); JSON and CSV output; debug visualisations.
+
+## Installation
+
+Python 3.10 or newer.
+
+```bash
+pip install spartan-tables                  # core: born-digital PDFs
+pip install "spartan-tables[ocr]"           # + Tesseract OCR for scanned pages
+pip install "spartan-tables[all]"           # + OCR, REST API, local LLM, memory stats
+```
+
+Extras: `ocr`, `api`, `llm`, `azure`, `metrics`, `all`. The package is imported as `spartan`.
+
+From source (for development):
+
+```bash
+git clone https://github.com/shlok-n/spartan.git spartan && cd spartan
+python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
+pip install -e ".[all,dev]"                           # or: pip install -r requirements.txt
+```
+
+Optional components:
+
+| Component | Needed for | Install |
+|---|---|---|
+| Tesseract binary | scanned pages / `--text-source ocr` | Windows: `winget install UB-Mannheim.TesseractOCR` · macOS: `brew install tesseract` · Debian/Ubuntu: `apt install tesseract-ocr` |
+| Ollama + a model | `--pcn` with LLM gap-filling | <https://ollama.com>, e.g. `ollama pull gemma4` |
+| Azure AI Vision / Azure OpenAI | optional cloud OCR / LLM | your own Azure resources, `pip install -e ".[azure]"` |
+
+Born-digital PDFs need neither OCR nor an LLM.
+
+## Configuration and credentials
+
+Runtime settings are read from environment variables prefixed with `SPARTAN_`, or from a
+`.env` file in the working directory:
+
+```bash
+cp .env.example .env      # then edit .env and add your own values
+```
+
+**The repository contains no credentials.** Cloud services are optional; to use them, put
+your own endpoint and key in `.env` (git-ignored). For example:
+
+```ini
+SPARTAN_AZURE_VISION_ENDPOINT=https://<your-resource>.cognitiveservices.azure.com/
+SPARTAN_AZURE_VISION_KEY=<your-azure-vision-key>
+```
+
+If a selected service is not configured, SPARTAN logs a warning explaining which variables
+to set and continues without it. See `.env.example` for every setting.
+
+## OCR engines
+
+OCR is only needed for pages without a text layer (scans) or with `--text-source ocr`.
+Choose the engine with `--ocr-engine` (or `PipelineConfig(ocr_engine=...)`):
+
+| Engine | Runs | Setup |
+|---|---|---|
+| `tesseract` (default) | offline | install the Tesseract binary; set `SPARTAN_TESSERACT_CMD` if it is not on `PATH` |
+| `azure` | Azure AI Vision Read API | `pip install -e ".[azure]"`, set `SPARTAN_AZURE_VISION_ENDPOINT` and `SPARTAN_AZURE_VISION_KEY` |
+| `none` | — | text layer only; scanned tables are detected but their cells stay empty |
+| `package.module:Class` | anything | your own engine, see below |
+
+`--ocr-scope` controls how OCR is applied: `page` (one call per page, words assigned to cells
+geometrically; the default) or the paper's cell strategies `single`, `multi` (cells stacked in
+chunks of `--chunk-size` with the row-separator image) and `auto`.
+
+### Using another OCR engine
+
+Any object with a `name`, `image_to_words(image)` and `image_to_text(image)` works (`image` is
+a BGR NumPy array). [`examples/custom_ocr_engine.py`](https://github.com/shlok-n/spartan/blob/main/examples/custom_ocr_engine.py) wraps
+EasyOCR in about 40 lines. Use it without modifying the package:
+
+```bash
+python -m spartan --file scan.pdf --ocr-engine examples.custom_ocr_engine:EasyOcrEngine
+```
+
+or register it under a short name from Python:
+
+```python
+from spartan import PipelineConfig, process_pdf
+from spartan.ocr import register_ocr_engine
+from examples.custom_ocr_engine import EasyOcrEngine
+
+register_ocr_engine("easyocr", EasyOcrEngine)
+document = process_pdf("scan.pdf", PipelineConfig(ocr_engine="easyocr"))
+```
+
+## Quick start
+
+```bash
+# Standard profile (pre-processing + region segmentation)
+python -m spartan --file notice.pdf
+
+# Every PDF in a folder, with CSV export and debug images
+python -m spartan --file ./pdfs --csv --save
+
+# Multi-column documents (e.g. scientific papers)
+python -m spartan --file paper.pdf --preprocess --column --region
+
+# Map to the PCN schema (Ollama by default, deterministic only with --llm-provider none)
+python -m spartan --file notice.pdf --pcn
+```
+
+Results are written to `output/<document>/<document>.json` (plus `csv/table_<n>.csv` and
+`<document>_pcn.json` when requested).
+
+## Using SPARTAN in your code
+
+```python
+import spartan
+
+document = spartan.process_pdf("notice.pdf")
+for table in document["tables"]:
+    print(table["title"], table["columns"], len(table["rows"]))
+```
+
+Tables load directly into pandas:
+
+```python
+import pandas as pd
+
+frames = [pd.DataFrame(t["rows"], columns=t["columns"]) for t in document["tables"]]
+```
+
+Options are set with `PipelineConfig` (all fields have sensible defaults):
+
+```python
+from spartan import PipelineConfig, SpartanPipeline, TextSourceMode
+
+config = PipelineConfig(
+    column_detection=True,               # multi-column layouts
+    text_source=TextSourceMode.AUTO,     # PDF text layer, OCR only where needed
+    ocr_engine="tesseract",              # or "azure", "none", "package.module:Class"
+    dpi=300,
+)
+pipeline = SpartanPipeline(config)       # reuse one pipeline for many files
+for path in ["a.pdf", "b.pdf"]:
+    document = pipeline.process(path)
+```
+
+`process_pdf` returns a JSON-serialisable `dict` (format below). `SpartanPipeline.process`
+also includes the internal `Table` objects under `document["_tables"]`, and
+`SpartanPipeline.process_and_save` writes JSON/CSV files like the CLI does.
+
+## CLI reference (paper Table 2)
+
+| Flag | Stage | Description |
+|---|---|---|
+| `--preprocess` | Pre-processing | grayscale, adaptive thresholding, fixed-DPI normalisation |
+| `--column` | Column detection | split multi-column layouts at vertical whitespace gutters |
+| `--region` | Region segmentation | table detection; pre-/inter-/post-text segmentation |
+| *(implicit)* | Table parsing | LSD line detection, grid and spanning-cell recovery |
+| `--text-source` | Data curation | `auto` (text layer, OCR fallback), `pdf`, `ocr` |
+| `--ocr-engine` | Data curation | `tesseract` (default), `azure`, `none` |
+| `--ocr-scope` / `--chunk-size` | Data curation | `page` (OCR once, assign words) or the paper's `single` / `multi` / `auto` cell strategies |
+| *(implicit)* | Post-processing | cross-page merging, header/footer de-duplication |
+| `--pcn`, `--llm-provider`, `--llm-model` | LLM hook | optional PCN schema mapping |
+| `--save`, `--csv`, `--stdout`, `--edge-detection`, `--dpi`, `--output` | Output | intermediate artefacts, CSV, JSON to stdout, Canny map, resolution, folder |
+
+As in the paper, a step runs only when its flag is given. Without any step flag, the
+standard profile `--preprocess --region` (the evaluation configuration) is used.
+
+## Output format
+
+```jsonc
+{
+  "document_name": "Analog Devices",
+  "tables": [{
+    "id": "table_2",
+    "pages": [1, 2],                      // merged across the page break
+    "type": "normal",                     // normal | nested | key_value
+    "title": "PDN 23_0061 Discontinued Parts Material List",
+    "title_source": "table",              // table | context | none
+    "columns": ["Model", "Product Family", "Replacement Part", "Pin To Pin Compatible", "Comments"],
+    "header_rows": 1,
+    "rows": [["ADAU1328BSTZ", "ADAU1328", "AD1938WBSTZ", "Yes", "None"], ...],
+    "merged_cells": [{"row": 4, "col": 2, "row_span": 2, "col_span": 1, "merge_type": "row", "text": "..."}],
+    "pairs": [...],                       // key_value tables only
+    "column_widths": [...], "detection_method": "boundary", "text_source": "pdf_text_layer",
+    "sources": [{"page": 1, "segment": "table_2", "bbox": [...]}, ...]
+  }],
+  "text": [{"page": 1, "id": "pre_text", "kind": "pre_text", "text": "...", "boilerplate": false}],
+  "full_text": "...",
+  "pages": [...],                         // per-page segments in reading order, timings
+  "processing": {"pages": 2, "seconds": 2.4, "seconds_per_page": 1.2, "peak_rss_mb": 185.0},
+  "pcn": {...}                            // only with --pcn
+}
+```
+
+Merged (spanning) cells are expanded: their value is repeated in every row and column
+they cover, as described in the paper, and listed in `merged_cells`.
+
+## REST API
+
+```bash
+uvicorn spartan.api:app --host 127.0.0.1 --port 8000     # or: python -m spartan.api
+```
+
+| Method | Path | Returns |
+|---|---|---|
+| `GET` | `/health` | status and version |
+| `POST` | `/extract?column_detection=false&pcn=false` | full document JSON |
+| `POST` | `/upload-pdf` | PCN schema (used by the SPARTAN front-end) |
+
+Uploads must be PDFs (checked by content) and are limited to `SPARTAN_MAX_UPLOAD_MB`.
+Errors are returned as `{"error": "..."}`.
+
+## PCN schema mapping
+
+`spartan/assets/fixed_schema.json` defines the output fields and
+`spartan/assets/configuration.json` lists example labels for each field. Mapping order:
+
+1. Table columns are matched to the table fields (`Impacted_Part`, `Replacement_Part`,
+   `Pin_to_Pin_Match`). Part fields also require the column content to look like part
+   numbers. Repeated column groups (`EOL Devices | Replacements | EOL Devices | …`) are
+   unrolled.
+2. Tables without matching columns, but with a title such as "Affected Part Numbers",
+   contribute all their cells to `Impacted_Part`.
+3. Scalar fields come from key-value tables, matching table columns and `Label: value`
+   lines. Dates must be parseable and are normalised to `YYYY-MM-DD`.
+4. Only what is still missing is sent to the LLM, with JSON-schema constrained output at
+   temperature 0.
+
+How each value was obtained is recorded in `document["pcn_provenance"]`.
+
+## Architecture
+
+| Module | Paper section |
+|---|---|
+| `pre_processing.py` | Pre-processing |
+| `column_detection.py` | Column detection |
+| `region_segmentation.py` | Region segmentation: crop, boundary detector, text detector, unification, segmentation |
+| `table_parser.py` | Table parsing (LSD, line merging, grid / spanning cells, refinement) |
+| `data_curation.py` | Data curation (table types, nested headers, merged cells, OCR correction, per-page merge) |
+| `text_extraction.py`, `ocr.py` | Text layer, OCR engines, single/multi-cell OCR, row-separator chunking |
+| `post_processing.py`, `table_merging.py` | Post-processing (cross-page merge, boilerplate removal) |
+| `plugins/pcn.py`, `plugins/llm.py` | LLM hook |
+| `config.py` | All heuristic thresholds (defined at 300 DPI and scaled automatically) and environment settings |
+| `pipeline.py`, `cli.py`, `api.py`, `export.py`, `visualization.py` | Orchestration and interfaces |
+
+## Testing
+
+```bash
+python -m pytest
+```
+
+Unit tests use synthetic images and need no external services. End-to-end tests run on real
+PCN/PDN PDFs from `../Dataset` (or the folder in `SPARTAN_DATASET_DIR`) and are skipped when
+it is absent; the documents are third-party and are not distributed with this repository.
+
+## Performance (23 PCN/PDN PDFs, ~90 pages, CPU only)
+
+About 0.3–1.5 s per page (the first pages include start-up), and about 280 MB peak memory,
+without OCR. Measured on the development laptop. The LLM step adds roughly a minute per
+document with an 8B model on CPU.
+
+## Known limitations
+
+- Scanned pages need the Tesseract binary; without it, tables on such pages are detected
+  but their cells stay empty (a warning is logged).
+- Heuristics assume roughly upright pages; heavily skewed scans are not de-skewed.
+- Word splitting relies on glyph geometry; PDFs that break a text run in the middle of a
+  word can still produce a stray split (e.g. "Withdrawa l").
+- Small borderless blocks whose first row is a set of `Label:` headers can be misread as
+  key-value tables.
+- The borderless detector needs at least three aligned rows and two columns.
+
+## Contributing
+
+Contributions of any size are welcome, and you don't need permission to start:
+
+- **Found a table SPARTAN gets wrong?** [Open an issue](https://github.com/shlok-n/spartan/issues/new/choose) with the
+  document or a screenshot. Hard examples are the most valuable contribution.
+- **Want to code?** Fork the repository, make your change with a test, and open a pull
+  request. [CONTRIBUTING.md](https://github.com/shlok-n/spartan/blob/main/CONTRIBUTING.md) explains the setup and lists open tasks, from
+  small fixes to de-skewing and benchmark evaluation.
+- **Using SPARTAN somewhere?** Let us know in an issue or discussion; it helps prioritise.
+
+Please follow the [Code of Conduct](https://github.com/shlok-n/spartan/blob/main/CODE_OF_CONDUCT.md). Report security issues privately
+as described in [SECURITY.md](https://github.com/shlok-n/spartan/blob/main/SECURITY.md).
+
+## License
+
+Released under the [MIT License](https://github.com/shlok-n/spartan/blob/main/LICENSE).
+
+## Citation
+
+If you use SPARTAN in your work, please cite the paper (see also [`CITATION.cff`](https://github.com/shlok-n/spartan/blob/main/CITATION.cff)):
+
+```bibtex
+@article{nandurbarkar2026spartan,
+  title   = {{SPARTAN}: automated table detection and extraction from documents using
+             advanced {OpenCV} heuristics and {OCR} techniques},
+  author  = {Nandurbarkar, S. and Chaudhari, A. Y. and Mulla, R.},
+  journal = {Scientific Reports},
+  volume  = {16},
+  pages   = {14954},
+  year    = {2026},
+  doi     = {10.1038/s41598-026-44325-7}
+}
+```