super-document-processor 0.1.3__tar.gz

super_document_processor-0.1.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Cinderfox
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

super_document_processor-0.1.3/PKG-INFO

@@ -0,0 +1,376 @@
+Metadata-Version: 2.4
+Name: super-document-processor
+Version: 0.1.3
+Summary: High-fidelity multi-format document parsing and chunking engine for RAG pipelines
+Author: Cinderfox
+License: MIT
+Project-URL: Homepage, https://github.com/Cinderfox/Super-Document-Processor
+Project-URL: Repository, https://github.com/Cinderfox/Super-Document-Processor
+Keywords: document-processing,rag,pdf,docx,pptx,xlsx,multimodal,vlm,chunking,parsing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: General
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyMuPDF==1.28.2
+Requires-Dist: python-docx==1.2.0
+Requires-Dist: python-pptx==1.0.2
+Requires-Dist: extract-msg==0.56.1
+Requires-Dist: lxml==6.1.3
+Requires-Dist: aspose-slides==26.7.0
+Requires-Dist: python-dotenv==1.2.3
+Requires-Dist: fastapi==0.141.1
+Requires-Dist: uvicorn==0.52.4
+Requires-Dist: python-multipart==0.0.32
+Provides-Extra: dev
+Requires-Dist: build>=1.0.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Dynamic: license-file
+
+# Super Document Processor
+
+A high-fidelity, multi-format document parsing and chunking engine tailored for Retrieval-Augmented Generation (RAG) pipelines. It extracts structured text, preserves tables, optionally transcribes embedded charts and images with Vision LLMs, and outputs boundary-aligned chunks.
+
+Available as a **Python library**, **CLI tool**, and **FastAPI microservice** (with Docker support).
+
+---
+
+## Supported Formats
+
+| Category | Formats | Features Extracted |
+|---|---|---|
+| **Documents** | `.pdf`, `.docx` | Headings, paragraphs, markdown/HTML tables, hyperlinks, embedded images |
+| **Presentations** | `.pptx` | Slide text, shapes, notes, multi-table separation, smart-art, charts |
+| **Emails** | `.msg` | Headers (From, To, Subject, Date) and formatted email body |
+| **Images** | `.jpg`, `.jpeg`, `.png`, `.jfif` | Full VLM vision analysis and table transcription |
+| **Code & Text** | `.txt`, `.md`, `.json`, `.csv`, `.py`, `.js`, `.ts`, `.html`, etc. | Natural paragraph and token-bounded chunking |
+
+---
+
+## Quickstart
+
+### 1. Local Setup
+```bash
+git clone https://github.com/Cinderfox/Super-Document-Processor.git
+cd Super-Document-Processor
+
+# Setup environment
+python -m venv .venv
+source .venv/bin/activate  # On Windows: .\.venv\Scripts\activate
+pip install -r requirements.txt
+
+# Configure environment (optional, required only for Vision LLMs)
+cp .env.example .env
+```
+
+### 2. Docker Setup
+```bash
+docker build -t super-document-processor:latest .
+docker run -d --name doc-processor -p 8000:8000 -v ${PWD}/data:/data super-document-processor:latest
+```
+
+---
+
+## 1. Python Library Usage
+
+```python
+from super_document_processor import process_file
+
+# 1. Standard text & table extraction (zero LLM calls, ultra-fast)
+chunks = process_file("annual_report.pdf")
+
+# 2. Extract with Vision LLM (describes embedded charts, diagrams, and photos)
+chunks_with_vision = process_file(
+    file_path="pitch_deck.pptx",
+    process_images=True,
+    model="google-gemini-3-flash"  # e.g. "gpt-4o", "google-gemini-3-flash", etc.
+)
+
+for chunk in chunks:
+    print(f"--- Chunk {chunk['chunkIndex']} ({chunk['filename']}) ---")
+    print(chunk["content"])
+```
+
+### Chunk Data Structure
+```json
+[
+  {
+    "content": "==================== Slide 1 ====================\nSlide Text:\n- Performance Scorecard...",
+    "chunkIndex": 1,
+    "filename": "pitch_deck.pptx"
+  }
+]
+```
+
+### Direct Chunker Access & Default Limits
+Text-based document processors (`.docx`, `.msg`, `.txt` etc) use `ChunkCollector` for token-safe output partitioning:
+- **Default Max Size**: `6,000` characters per chunk
+- **Default Overlap**: `0` characters (zero duplicated text across chunks)
+
+```python
+from super_document_processor.utils.chunker import collector, chunk_text
+
+# Using standalone utility with custom values:
+chunks = chunk_text(text="Your long text...", chunk_size=6000, overlap=200)
+
+# Or collecting and retrieving chunked texts:
+collector.add("First section text...")
+chunks = collector.get_chunked_texts(max_size=6000, overlap=200)
+collector.clear()
+```
+
+### PPTX Slide Rendering & Image Conversion
+Export PowerPoint slides directly to SVGs or PNGs:
+
+```python
+from utils.pptx_converter import convert_pptx, create_pptx_render_zip
+
+# 1. Convert all slides to SVGs (returns list of clean SVG strings)
+res = convert_pptx("presentation.pptx", format="svg")
+for s in res["slides"]:
+    print(f"Slide {s['slide']}: {s['svg'][:80]}...")
+
+# 2. Convert specific slide to high-res PNG bytes
+res_png = convert_pptx("presentation.pptx", format="png", slide=1, scale=2.0)
+with open("slide_1.png", "wb") as f:
+    f.write(res_png["raw_bytes"])
+
+# 3. Create a downloadable ZIP containing all rendered slides
+zip_bytes = create_pptx_render_zip("presentation.pptx", format="png")
+with open("presentation_pngs.zip", "wb") as f:
+    f.write(zip_bytes)
+```
+
+---
+
+## 2. CLI Usage
+
+```bash
+# Basic extraction to terminal (stdout)
+python main.py document.pdf
+
+# Save raw JSON chunks to a file
+python main.py report.docx --json -o output.json
+
+# Process embedded images using a Vision Model
+python main.py slides.pptx --process-images --model google-gemini-3-flash --json -o output.json
+
+# Convert all PPTX slides to clean SVG files in a directory
+python main.py slides.pptx --convert-pptx --format svg -o ./output_svgs
+
+# Convert specific slide to PNG
+python main.py slides.pptx --convert-pptx --format png --slide 1 -o slide_1.png
+
+# Convert PPTX and package all slides into a ZIP archive
+python main.py slides.pptx --convert-pptx --format png --as-zip -o slides.zip
+
+# Start the REST API server
+python main.py --serve --port 8000
+```
+
+| Flag | Short | Default | Description |
+|---|---|---|---|
+| `file` | | *None* | Path to document or image file |
+| `--json` | | `False` | Output as raw JSON chunks or JSON slide array |
+| `--output` | `-o` | `stdout` | Destination file path or directory for output |
+| `--convert-pptx` | `-c` | `False` | Convert PPTX slides to SVG or PNG images |
+| `--format` | `-f` | `svg` | Target conversion format (`svg` or `png`) |
+| `--slide` | | *All* | Specific 1-based slide index to convert |
+| `--scale` | | `2.0` | Resolution scale factor for PPTX rendering |
+| `--as-zip` | | `False` | Bundle all converted slides into a single `.zip` file |
+| `--process-images` | | `False` | Enable AI vision processing for images/tables |
+| `--model` | `-m` | *None* | Vision model name (required if `--process-images` is set) |
+| `--serve` | `-s` | `False` | Start the FastAPI HTTP microservice |
+| `--port` | `-p` | `8000` | Server listening port |
+
+---
+
+## 3. REST API Microservice
+
+Start the server with `python main.py --serve --port 8000` or via Docker.
+- **Swagger Docs**: `http://localhost:8000/docs`
+- **Health Check**: `http://localhost:8000/health`
+
+### 1. Document Parsing & Chunking: `POST /process`
+
+Accepts **either** a file upload (`multipart/form-data`) **or** a local/volume file path (`application/json`). Providing both or neither returns an HTTP 400 error.
+
+#### Option A: File Upload (Multipart Form)
+```bash
+# Basic upload
+curl -X POST "http://localhost:8000/process" \
+     -F "file=@/path/to/contract.pdf"
+
+# With Vision LLM processing
+curl -X POST "http://localhost:8000/process?process_images=true&model=google-gemini-3-flash" \
+     -F "file=@/path/to/presentation.pptx"
+```
+
+#### Option B: Local File Path or Mounted Volume Path (JSON)
+```bash
+# Running locally:
+curl -X POST "http://localhost:8000/process" \
+     -H "Content-Type: application/json" \
+     -d '{"file_path": "./test_docs/report.pdf"}'
+
+# Running in Docker:
+curl -X POST "http://localhost:8000/process" \
+     -H "Content-Type: application/json" \
+     -d '{"file_path": "/data/report.pdf", "process_images": true, "model": "gpt-4o"}'
+```
+
+---
+
+### 2. PPTX to SVG / PNG Conversion: `POST /convert/pptx`
+
+Renders PowerPoint slides into SVG or PNG images.
+
+#### Stream Single Slide Directly as Image (`image/svg+xml` or `image/png`)
+Ideal for embedding directly in `<img>` tags or displaying in browsers:
+```bash
+# Get Slide 1 as direct SVG
+curl -X POST "http://localhost:8000/convert/pptx?slide=1&format=svg" \
+     -F "file=@presentation.pptx" > slide_1.svg
+
+# Get Slide 2 as direct high-res PNG
+curl -X POST "http://localhost:8000/convert/pptx?slide=2&format=png&scale=2.0" \
+     -F "file=@presentation.pptx" > slide_2.png
+```
+
+#### Download All Slides as a ZIP Archive
+```bash
+curl -X POST "http://localhost:8000/convert/pptx?format=png&as_zip=true" \
+     -F "file=@presentation.pptx" -o presentation_slides.zip
+```
+
+#### Convert Mounted Volume Path via JSON Body
+```bash
+curl -X POST "http://localhost:8000/convert/pptx" \
+     -H "Content-Type: application/json" \
+     -d '{"file_path": "/data/presentation.pptx", "format": "svg"}'
+```
+
+#### Python Client Example
+```python
+import requests
+
+url = "http://localhost:8000/convert/pptx"
+
+# 1. Download Slide 1 directly as image
+with open("pitch_deck.pptx", "rb") as f:
+    resp = requests.post(url, files={"file": f}, params={"slide": 1, "format": "png"})
+    with open("slide_1.png", "wb") as out:
+        out.write(resp.content)
+
+# 2. Get all slides as JSON payload (contains SVGs or base64 PNGs)
+with open("pitch_deck.pptx", "rb") as f:
+    resp = requests.post(url, files={"file": f}, params={"format": "svg", "response_type": "json"})
+    data = resp.json()
+    print(f"Rendered {data['total_slides']} slides.")
+```
+
+---
+
+### 3. Async Processing: `POST /process-async` & `GET /jobs/{job_id}`
+
+For large files or heavy multimodal extraction, submit jobs asynchronously to prevent client gateway timeouts. No Redis required (backed by persistent `/data/jobs` storage).
+
+#### 1. Submit Job (`POST /process-async`)
+```bash
+curl -X POST "http://localhost:8000/process-async" \
+     -F "file=@large_document.pdf"
+```
+```json
+{
+  "job_id": "e8d7a12b-34c5-4f67-8901-abcdef123456",
+  "filename": "large_document.pdf",
+  "status": "processing",
+  "poll_url": "/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"
+}
+```
+
+#### 2. Poll Status & Fetch Result (`GET /jobs/{job_id}`)
+```bash
+curl "http://localhost:8000/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"
+```
+- **In Progress**: `{"status": "processing", "progress": 0}`
+- **Completed**: Returns the full JSON payload (`"status": "completed"`, `total_chunks`, `chunks`). The job file is automatically deleted from disk upon successful retrieval.
+
+---
+
+## 4. Docker Deployment
+
+The container image includes all rendering dependencies (`libicu`, `ca-certificates`, etc.).
+
+### Run as an API Server
+```bash
+# Port 8000 exposed, local ./data folder mounted to /data in container
+docker run -d \
+  --name doc-processor \
+  -p 8000:8000 \
+  --env-file .env \
+  -v ${PWD}/data:/data \
+  super-document-processor:latest
+```
+
+### Run as One-Off CLI Tool & Save Output to Host
+Files created inside Docker default to the container's isolated layer. To save the output JSON directly on your host machine, use either **Volume Mounting** or **Shell Redirection**:
+
+```bash
+# Method 1: Mount folder and write with -o (appears in your host folder)
+docker run --rm -v ${PWD}:/data super-document-processor:latest \
+  python main.py /data/report.pdf --json -o /data/output.json
+
+# Method 2: Stream stdout directly to host file (works with docker run or docker exec)
+docker run --rm -v ${PWD}:/data super-document-processor:latest \
+  python main.py /data/report.pdf --json > output.json
+
+# Method 3: Run against an already active container
+docker exec doc-processor python main.py /data/report.pdf --json > output.json
+```
+
+### Docker Compose
+```yaml
+version: "3.8"
+
+services:
+  doc-processor:
+    image: super-document-processor:latest
+    container_name: doc-processor
+    restart: unless-stopped
+    ports:
+      - "8000:8000"
+    env_file:
+      - .env
+    volumes:
+      - ./data:/data
+```
+
+```bash
+docker compose up -d
+```
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `PORT` | `8000` | HTTP port when running in server mode |
+| `OPENAI_API_KEY` | *None* | API key for Vision LLM extraction |
+| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Custom gateway URL (LiteLLM, Ollama, Gemini, Groq, etc.) |
+
+---
+
+## License
+
+[MIT License](LICENSE) © [Cinderfox](https://github.com/Cinderfox)