rc-docparser 0.2.0__tar.gz

rc_docparser-0.2.0/.gitignore

@@ -0,0 +1,37 @@
+# --- secrets / env ---
+.env
+.env.*
+!.env.example
+
+# --- Python ---
+.venv/
+venv/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.coverage.*
+htmlcov/
+
+# --- build artefacts ---
+build/
+dist/
+*.whl
+*.tar.gz
+
+# --- macOS ---
+.DS_Store
+
+# --- editor ---
+.vscode/
+.idea/
+*.swp
+
+# --- pipeline cache ---
+.cache/

rc_docparser-0.2.0/CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+All notable changes to **docparser** are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/) and the project
+follows [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-06-16
+### Added
+- PPTX parser (extra `[pptx]`): walks slides in order; emits per-slide
+  headings, bulleted text, tables, images, and speaker notes; image captioning.
+- Plain-text and Markdown parser (core): `.txt` / `.md` into structured blocks
+  (headings, list items, code fences, paragraphs).
+- CSV/TSV parser (core): delimiter sniffing, header detection, Markdown table
+  + per-row JSON records.
+- EPUB parser (extra `[epub]`): spine-ordered chapter extraction with a
+  BeautifulSoup structural walk and embedded-image captioning.
+- Pluggable high-fidelity PDF backends via `parse_pdf(backend=...)` and
+  `docparser.pdf_backends`: `pymupdf4llm`, `docling`, `marker` (each an opt-in
+  extra; Marker/PyMuPDF4LLM licenses noted).
+- OCR for scanned/low-text PDFs via `parse_pdf(ocr="off|auto|force")` and an
+  image OCR helper (`docparser.ocr`, extra `[ocr]` using `rapidocr-onnxruntime`).
+- Better PDF tables via `parse_pdf(extract_tables=True)` using `pdfplumber`
+  (extra `[tables]`), emitting real `table` blocks.
+- Multi-provider captioning: `caption_image(provider=...)` presets for
+  `openrouter` / `openai` / `gemini` / `local` (any OpenAI-compatible endpoint),
+  plus a fully-local `transformers` backend (`docparser.localvlm`, extra
+  `[localvlm]`).
+- CLI flags: `--vlm-provider`, `--vlm-model`, `--pdf-backend`, `--ocr`,
+  `--pdf-tables`; `run_all`/`parse_path` thread these options through.
+- Typing: PEP 561 `py.typed` marker and a `mypy` configuration.
+- GitHub Actions: CI (lint + mypy + tests across Python 3.10-3.12 + build) and
+  a tag-triggered PyPI Trusted-Publishing workflow.
+
+### Changed
+- `parse_path` and `run_all` accept the new PDF and captioning options;
+  non-PDF parsers ignore PDF-only keyword arguments.
+
+## [0.1.0] - 2026-06-12
+### Added
+- Initial public release.
+- DOCX parser: walks the document body in order; preserves headings, lists,
+  paragraphs, tables, and inline images; associates figure captions and
+  surrounding context with each image.
+- XLSX parser: traverses every sheet, every row, every column; preserves cell
+  types, formulas, hyperlinks, comments, merged ranges, frozen panes, and
+  embedded images.
+- PDF parser (extra `[pdf]`): page-by-page text + image extraction via PyMuPDF;
+  best-effort heading detection from font sizing.
+- HTML parser (extra `[html]`): article-grade text extraction via trafilatura
+  with a BeautifulSoup-based fallback that preserves headings, lists, and
+  tables.
+- Image semantic captioner (extra `[vlm]`): OpenRouter-backed VLM client with
+  on-disk caching keyed by image SHA-1; returns
+  `{caption, description, visible_text, tags, image_kind, domain_relevance}`.
+- `docparser` CLI: `parse` (single file), `parse-all` (directory walk),
+  `version`.
+- `WorkspaceLayout` dataclass and `parse_path` dispatcher for library use.

rc_docparser-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Research Commons
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rc_docparser-0.2.0/PKG-INFO

@@ -0,0 +1,344 @@
+Metadata-Version: 2.4
+Name: rc-docparser
+Version: 0.2.0
+Summary: Convert research literature (.docx, .xlsx, .pdf, .html, .pptx, .epub, .txt, .md, .csv) into structured Markdown + JSON corpora, with optional VLM image semantic captioning.
+Project-URL: Homepage, https://github.com/Research-Commons/docparser
+Project-URL: Repository, https://github.com/Research-Commons/docparser
+Project-URL: Issues, https://github.com/Research-Commons/docparser/issues
+Project-URL: Changelog, https://github.com/Research-Commons/docparser/blob/main/CHANGELOG.md
+Author-email: Research Commons <shubhankitsingh@researchcommons.ai>
+License: MIT License
+        
+        Copyright (c) 2026 Research Commons
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: corpus,csv,docx,epub,html,literature,markdown,ocr,parser,pdf,pptx,rag,vlm,xlsx
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Text Processing :: Markup
+Requires-Python: >=3.10
+Requires-Dist: lxml>=5.3.0
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: pillow>=10.4.0
+Requires-Dist: python-docx>=1.1.2
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: tqdm>=4.66.5
+Provides-Extra: all
+Requires-Dist: beautifulsoup4>=4.12.0; extra == 'all'
+Requires-Dist: ebooklib>=0.18; extra == 'all'
+Requires-Dist: numpy>=1.24.0; extra == 'all'
+Requires-Dist: pdfplumber>=0.11.0; extra == 'all'
+Requires-Dist: pymupdf>=1.24.0; extra == 'all'
+Requires-Dist: python-pptx>=1.0.0; extra == 'all'
+Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'all'
+Requires-Dist: requests>=2.32.3; extra == 'all'
+Requires-Dist: trafilatura>=1.12.0; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == 'dev'
+Requires-Dist: ebooklib>=0.18; extra == 'dev'
+Requires-Dist: mypy>=1.10.0; extra == 'dev'
+Requires-Dist: pandas>=2.2.0; extra == 'dev'
+Requires-Dist: pdfplumber>=0.11.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: python-pptx>=1.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.6.0; extra == 'dev'
+Requires-Dist: twine>=5.1.0; extra == 'dev'
+Provides-Extra: docling
+Requires-Dist: docling>=2.0.0; extra == 'docling'
+Provides-Extra: epub
+Requires-Dist: beautifulsoup4>=4.12.0; extra == 'epub'
+Requires-Dist: ebooklib>=0.18; extra == 'epub'
+Provides-Extra: html
+Requires-Dist: beautifulsoup4>=4.12.0; extra == 'html'
+Requires-Dist: trafilatura>=1.12.0; extra == 'html'
+Provides-Extra: localvlm
+Requires-Dist: pillow>=10.4.0; extra == 'localvlm'
+Requires-Dist: torch>=2.2.0; extra == 'localvlm'
+Requires-Dist: transformers>=4.40.0; extra == 'localvlm'
+Provides-Extra: marker
+Requires-Dist: marker-pdf>=1.0.0; extra == 'marker'
+Provides-Extra: ocr
+Requires-Dist: numpy>=1.24.0; extra == 'ocr'
+Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'ocr'
+Provides-Extra: pdf
+Requires-Dist: pymupdf>=1.24.0; extra == 'pdf'
+Provides-Extra: pptx
+Requires-Dist: python-pptx>=1.0.0; extra == 'pptx'
+Provides-Extra: pymupdf4llm
+Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pymupdf4llm'
+Provides-Extra: tables
+Requires-Dist: pdfplumber>=0.11.0; extra == 'tables'
+Provides-Extra: vlm
+Requires-Dist: requests>=2.32.3; extra == 'vlm'
+Description-Content-Type: text/markdown
+
+# docparser
+
+Convert research literature (`.docx`, `.xlsx`, `.pdf`, `.html`, `.pptx`,
+`.epub`, `.txt`, `.md`, `.csv`) into a clean, reproducible **Markdown + JSON**
+corpus, with optional **vision-language captioning** of every embedded figure
+via OpenRouter, OpenAI, Gemini, a local server, or a fully-local model.
+
+```text
+                              ┌────────────────┐
+  data/raw/*.docx             │   docparser    │       data/parsed/<slug>/
+  data/raw/*.xlsx             │  - parse_docx  │       document.md
+  data/raw/*.pdf      ─────►  │  - parse_xlsx  │ ────► document.json
+  data/raw/*.html             │  - parse_pdf   │
+  data/raw/*.pptx             │  - parse_html  │       data/assets/<slug>/
+  data/raw/*.epub             │  - parse_pptx  │       img-*.png
+  data/raw/*.txt|md|csv       │  - parse_epub  │
+                              │  - VLM caption │
+                              └────────────────┘
+```
+
+## Install
+
+```bash
+pip install docparser              # core: docx + xlsx + txt/md + csv/tsv
+pip install 'docparser[pdf]'       # + PyMuPDF for PDFs
+pip install 'docparser[html]'      # + trafilatura + bs4 for HTML
+pip install 'docparser[pptx]'      # + python-pptx for PowerPoint
+pip install 'docparser[epub]'      # + EbookLib + bs4 for EPUB
+pip install 'docparser[vlm]'       # + requests for API VLM captions
+pip install 'docparser[all]'       # everything above (recommended)
+```
+
+Higher-fidelity / heavier features are separate opt-in extras (so the core
+install stays small and MIT):
+
+```bash
+pip install 'docparser[tables]'       # + pdfplumber for PDF table extraction
+pip install 'docparser[ocr]'          # + rapidocr-onnxruntime for scanned PDFs
+pip install 'docparser[pymupdf4llm]'  # PyMuPDF4LLM PDF backend (AGPL/commercial)
+pip install 'docparser[docling]'      # IBM Docling PDF backend (MIT)
+pip install 'docparser[marker]'       # Datalab Marker PDF backend (GPL-3.0)
+pip install 'docparser[localvlm]'     # transformers/torch local captioning
+```
+
+`docparser` requires Python 3.10+.
+
+## Quick start (library)
+
+```python
+from docparser import WorkspaceLayout, run_all
+
+layout = WorkspaceLayout.under("./project")   # data/raw, data/parsed, data/assets, .cache
+layout.ensure()
+
+run_all(layout, use_vlm=False)                # parse everything in data/raw
+```
+
+For a single file:
+
+```python
+from docparser import parse_path, WorkspaceLayout
+
+layout = WorkspaceLayout.under(".")
+payload = parse_path("paper.pdf", layout)
+print(payload["stats"])
+```
+
+## Quick start (CLI)
+
+```bash
+# parse a single file
+docparser parse paper.pdf --workspace ./out --no-vlm
+
+# walk a whole directory
+docparser parse-all --workspace ./project --no-vlm
+
+# enable VLM captioning (requires OPENROUTER_API_KEY in env or .env)
+export OPENROUTER_API_KEY=sk-or-v1-...
+docparser parse-all --workspace ./project --max-images 50
+
+# higher-fidelity PDF: pick a backend, OCR scanned pages, extract tables
+docparser parse paper.pdf --pdf-backend docling --ocr auto --pdf-tables --no-vlm
+
+# caption with a different provider
+docparser parse-all --workspace ./project --vlm-provider openai --vlm-model gpt-4o-mini
+
+docparser version
+```
+
+## What gets captured
+
+### `.docx`
+- Walks the document body in **document order** (paragraphs + tables + drawings).
+- Preserves heading hierarchy (`section_path`) on every block.
+- Extracts every embedded image to `data/assets/<slug>/` with a stable
+  `img-<seq>-<sha10>.<ext>` name.
+- Detects figure/table captions (style `Caption` or text matching
+  `Figure 1: …` / `Fig. 1.` / `Table 1.`) and associates the caption with the
+  preceding image.
+- Captures `context_before` and `context_after` for every image so the VLM has
+  document-grounded context.
+
+### `.xlsx`
+- Iterates **every sheet, every row, every column**.
+- For each cell stores: address, row/col indices, value, openpyxl `data_type`,
+  `number_format`, `hyperlink`, `comment`, and the **formula** (from a second
+  pass with `data_only=False`).
+- Stores `merged_ranges`, `frozen_panes`, and any embedded images.
+- Markdown rendering uses the first non-empty row as a header heuristic and
+  preserves multi-line cells with `<br>`.
+
+### `.pdf`
+- Page-by-page text extraction in reading order via PyMuPDF's blocks API.
+- Best-effort heading detection from font size (≥120% of the body-text median
+  promotes a line to a heading; bold flag tracked).
+- Embedded raster images extracted via `doc.extract_image(xref)`.
+- **Pluggable backends** (`backend="pymupdf4llm" | "docling" | "marker"`) route
+  conversion to a higher-fidelity engine; their Markdown is normalized into the
+  same block schema. Images are still extracted via PyMuPDF.
+- **OCR** (`ocr="auto" | "force"`, `[ocr]` extra) recognizes text on scanned /
+  low-text pages; OCR'd blocks carry `"ocr": true`.
+- **Tables** (`extract_tables=True`, `[tables]` extra) emit real `table` blocks
+  via `pdfplumber`.
+
+### `.html`
+- Article-grade body extraction via `trafilatura`.
+- Plus a **structural** BeautifulSoup walk that emits typed blocks
+  (`heading` / `paragraph` / `list` / `table` / `image`) so downstream RAG
+  layers can rely on the JSON.
+
+### `.pptx`
+- Walks slides in presentation order; each slide becomes a section.
+- Emits per-slide headings (slide title), bulleted text frames (with list
+  level), tables, pictures, and **speaker notes**.
+- Embedded pictures extracted and optionally captioned.
+
+### `.epub`
+- Walks the spine in reading order; per-chapter BeautifulSoup structural walk.
+- Captures metadata (title/author/language), headings, paragraphs, lists,
+  tables, and embedded images (resolved from the EPUB image manifest).
+
+### `.txt` / `.md` and `.csv` / `.tsv` (core, no extras)
+- Plain text is split into paragraph blocks; Markdown is passed through and
+  also decomposed into heading / list / code / paragraph blocks.
+- CSV/TSV: delimiter sniffing, header detection, a Markdown table, and one JSON
+  record per row.
+
+### Images (`[vlm]` extra)
+Each image is sent to a vision-language model (default provider OpenRouter,
+model `anthropic/claude-sonnet-4`) along with its surrounding caption +
+context. Any OpenAI-compatible provider works via `--vlm-provider`
+(`openrouter` / `openai` / `gemini` / `local`), or use a fully-local
+`transformers` model with `--vlm-provider transformers` (`[localvlm]` extra).
+The model returns a strict JSON object:
+
+```json
+{
+  "caption":           "one-sentence figure caption",
+  "description":       "2–5 sentence paragraph",
+  "visible_text":      "OCR-style transcription",
+  "tags":              ["world-model", "diagram", "..."],
+  "image_kind":        "diagram | plot | screenshot | photo | equation | table | ...",
+  "domain_relevance":  "how this relates to the document's topic"
+}
+```
+
+Results are cached on disk at `<cache_dir>/vlm/<model>/<sha1>.json`, keyed by
+**SHA-1 of the image bytes × model**, so re-runs are free until the source
+image bytes change.
+
+## Configuration (`.env`)
+
+| Var | Default | Purpose |
+| --- | --- | --- |
+| `DOCPARSER_VLM_PROVIDER` | `openrouter` | `openrouter` / `openai` / `gemini` / `local` |
+| `OPENROUTER_API_KEY` | _required for OpenRouter_ | OpenRouter key (`sk-or-...`) |
+| `OPENROUTER_VLM_MODEL` | `anthropic/claude-sonnet-4` | any vision-capable OpenRouter model |
+| `OPENROUTER_BASE_URL` | `https://openrouter.ai/api/v1` | override for a proxy |
+| `OPENROUTER_REFERER` / `OPENROUTER_TITLE` | repo URL / `docparser` | OpenRouter attribution headers |
+| `OPENAI_API_KEY` / `OPENAI_VLM_MODEL` | _required for OpenAI_ / `gpt-4o-mini` | OpenAI provider |
+| `GEMINI_API_KEY` / `GEMINI_VLM_MODEL` | _required for Gemini_ / `gemini-1.5-flash` | Gemini provider |
+| `DOCPARSER_VLM_BASE_URL` | `http://localhost:11434/v1` | base URL for the `local` provider |
+| `DOCPARSER_VLM_API_KEY` / `DOCPARSER_VLM_MODEL` | — / `llava` | key + model for the `local` provider |
+| `DOCPARSER_LOCAL_VLM_MODEL` | `Salesforce/blip-image-captioning-large` | model for the `transformers` backend |
+
+## API reference (highlights)
+
+- `WorkspaceLayout(raw_dir, parsed_dir, assets_dir, cache_dir)` —
+  dataclass describing where parser output lives. Use `.under(root)` for the
+  default `data/raw + data/parsed + data/assets + .cache` layout under a root.
+- `parse_docx(source, layout=None, *, captioner=None, write_outputs=True)` →
+  payload dict.
+- `parse_xlsx(source, layout=None, *, captioner=None, write_outputs=True)` →
+  payload dict.
+- `parse_pdf(source, layout=None, *, captioner=None, write_outputs=True, extract_images=True, backend="builtin", ocr="off", extract_tables=False)` →
+  payload dict. (requires `[pdf]`; backends/OCR/tables require their extras)
+- `parse_html(source, layout=None, *, captioner=None, write_outputs=True, use_trafilatura=True)` →
+  payload dict. `source` may be a path or `http(s)://` URL. (requires `[html]`)
+- `parse_pptx(source, layout=None, *, captioner=None, write_outputs=True)` →
+  payload dict. (requires `[pptx]`)
+- `parse_epub(source, layout=None, *, captioner=None, write_outputs=True)` →
+  payload dict. (requires `[epub]`)
+- `parse_text(source, layout=None, ...)` / `parse_csv(source, layout=None, ...)` —
+  core parsers for `.txt`/`.md` and `.csv`/`.tsv`.
+- `parse_path(source, layout=None, **kwargs)` — dispatches by extension;
+  PDF-only kwargs (`backend`, `ocr`, `extract_tables`) are forwarded to PDFs.
+- `run_all(layout, *, use_vlm=True, only=None, max_images=None, continue_on_error=False, vlm_provider=None, vlm_model=None, pdf_backend="builtin", ocr="off", extract_tables=False)` —
+  walks `layout.raw_dir`, parses everything supported, writes a top-level
+  `CORPUS.md` and `data/parsed/corpus.json`.
+- `caption_image(image_bytes, *, mime, doc_name, nearby_caption, context, provider=None, model=None, layout=None, ...)` →
+  `VLMResult`. (requires `[vlm]`)
+
+## Development
+
+```bash
+git clone https://github.com/Research-Commons/docparser
+cd docparser
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[all,dev]"
+pytest -ra
+ruff check src tests
+mypy
+python -m build           # produces dist/*.whl + *.tar.gz
+twine check dist/*
+```
+
+### Publishing
+
+CI runs lint + mypy + tests on Python 3.10-3.12 and builds the distribution on
+every push/PR (`.github/workflows/ci.yml`). Pushing a version tag (e.g.
+`v0.2.0`) triggers `.github/workflows/publish.yml`, which builds and uploads to
+PyPI via **Trusted Publishing** (OIDC, no stored token) — configure a trusted
+publisher for the project on PyPI first. To publish manually instead:
+
+```bash
+python -m build
+twine check dist/*
+twine upload dist/*
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).