llmPDF 0.7.0__tar.gz

llmpdf-0.7.0/LICENSE

@@ -0,0 +1,103 @@
+llmPDF Limited Use License 1.0
+
+Copyright (c) 2026 yechenyan. All rights reserved.
+
+1. Definitions
+
+"Software" means the llmPDF software distributed with this license, including
+its source code, object code, documentation, and original assets.
+
+"Unmodified Copy" means a complete copy of the Software whose contents have
+not been altered. Ordinary installation, execution, configuration through
+documented options, creation of bytecode or caches, and use through documented
+command-line or programming interfaces do not make a copy modified.
+
+2. License Grant
+
+Subject to this license, the copyright holder grants you a worldwide,
+royalty-free, non-exclusive license to:
+
+a. install, execute, and use Unmodified Copies for any purpose, including
+   commercial purposes;
+
+b. integrate an Unmodified Copy with other software through the Software's
+   documented command-line and programming interfaces; and
+
+c. reproduce and redistribute Unmodified Copies, either separately or together
+   with other software, provided that you comply with Section 3.
+
+3. Attribution and Notices
+
+Every redistributed copy must include this license and all copyright,
+attribution, and proprietary notices supplied with the Software.
+
+Any product, distribution, documentation, credits page, or other materials that
+accompany a redistributed copy must display the following notice in a
+reasonably prominent manner:
+
+    This product includes llmPDF, Copyright (c) 2026 yechenyan,
+    used under the llmPDF Limited Use License 1.0.
+
+If the distribution has no documentation or credits page, the notice must be
+provided in another reasonably prominent location accompanying the Software.
+
+4. Restrictions
+
+Except where applicable law expressly permits and does not allow the
+restriction to be waived, you may not:
+
+a. modify, adapt, translate, transform, or create derivative works of the
+   Software;
+
+b. distribute or make available a modified version or derivative work of the
+   Software;
+
+c. remove, obscure, or alter any copyright, attribution, license, or
+   proprietary notice;
+
+d. sublicense the Software or grant rights in it beyond those provided by this
+   license; or
+
+e. use the names, logos, or trademarks of llmPDF or the copyright holder to
+   imply endorsement, sponsorship, or affiliation.
+
+5. Ownership
+
+The Software is licensed, not sold. The copyright holder retains all rights,
+title, and interest in the Software except for the limited permissions expressly
+granted by this license. No patent or trademark rights are granted.
+
+6. Third-Party Components
+
+Third-party software and assets included with or required by the Software are
+licensed under their respective terms. This license applies only to the rights
+held by the llmPDF copyright holder and does not replace or limit any
+third-party license.
+
+7. Termination
+
+Your rights under this license terminate automatically if you breach its terms.
+Upon termination, you must stop using and redistributing the Software and
+destroy copies in your possession or control, except where retention is required
+by law. The copyright holder may reinstate your rights in writing.
+
+8. Disclaimer of Warranty
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE, TITLE, AND NON-INFRINGEMENT. YOU BEAR ALL RISK ARISING FROM USE OF THE
+SOFTWARE.
+
+9. Limitation of Liability
+
+TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE COPYRIGHT HOLDER WILL NOT BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, CONSEQUENTIAL, OR
+OTHER DAMAGES ARISING FROM OR RELATED TO THE SOFTWARE OR THIS LICENSE, EVEN IF
+ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+10. Severability
+
+If any provision of this license is held unenforceable, it will be limited or
+removed only to the minimum extent necessary, and the remaining provisions will
+remain in effect.
+

llmpdf-0.7.0/MANIFEST.in

@@ -0,0 +1 @@
+include SDK.md

llmpdf-0.7.0/PKG-INFO

@@ -0,0 +1,488 @@
+Metadata-Version: 2.4
+Name: llmPDF
+Version: 0.7.0
+Summary: Task-oriented PDF conversion and review toolkit
+Author-email: yechenyan <ccyechenyan@gmail.com>
+License-Expression: LicenseRef-llmPDF-Limited-Use-1.0
+Project-URL: Homepage, https://github.com/yechenyan/llmPDF
+Project-URL: Repository, https://github.com/yechenyan/llmPDF
+Project-URL: Issues, https://github.com/yechenyan/llmPDF/issues
+Keywords: pdf,markdown,docling,document-processing,llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Text Processing :: Markup
+Requires-Python: <3.15,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: docling<2.124,>=2.123
+Requires-Dist: pdfplumber<0.12,>=0.11
+Requires-Dist: Pillow<13,>=11
+Requires-Dist: pypdf<7,>=6
+Requires-Dist: PyYAML<7,>=6
+Provides-Extra: review
+Requires-Dist: fastapi<1,>=0.115; extra == "review"
+Requires-Dist: uvicorn<1,>=0.30; extra == "review"
+Dynamic: license-file
+
+# llmPDF
+
+Convert PDFs into Markdown that is suitable for reading, search, and downstream processing. The tool preserves document structure, extracts tables, and can analyze images and charts. A local review UI lets you compare the generated Markdown with the source PDF.
+
+Use `llmpdf convert` for normal operation. One command runs the complete conversion.
+
+## Requirements
+
+- Python 3.11–3.14
+- [uv](https://docs.astral.sh/uv/)
+- Poppler's `pdftoppm`
+- Node.js 20.6+
+- A working Codex/Pi login and network connection
+
+Install Poppler on macOS with Homebrew:
+
+```bash
+brew install poppler
+```
+
+On Ubuntu or Debian, install `poppler-utils`:
+
+```bash
+sudo apt-get install poppler-utils
+```
+
+The first run may download document-analysis models and can take longer than later runs.
+
+## Installation
+
+From the project directory, run:
+
+```bash
+uv sync --extra review --locked
+```
+
+This installs both the converter and the optional local review UI.
+
+Verify the installation:
+
+```bash
+uv run llmpdf --help
+uv run llmpdf convert --help
+```
+
+## Quick start
+
+```bash
+uv run llmpdf convert \
+  "/absolute/path/to/input.pdf" \
+  --output-dir "/absolute/path/to/results"
+```
+
+On success, the command prints JSON to standard output. It includes `output_markdown`, `output_dir`, token usage, and elapsed time.
+
+The default result directory is:
+
+```text
+<output-dir>/<UTC batch timestamp>/<PDF filename>/
+```
+
+Set a stable batch name when you need a predictable path or resumable retries:
+
+```bash
+uv run llmpdf convert \
+  "/absolute/path/to/input.pdf" \
+  --output-dir "/absolute/path/to/results" \
+  --batch-id "edis-2024"
+```
+
+After a failure, rerun with the same PDF, `--output-dir`, and `--batch-id`. Completed results that remain valid are reused. Use `--force` only when every stage must run again.
+
+To process selected pages while retaining their original PDF page numbers:
+
+```bash
+uv run llmpdf convert \
+  "/absolute/path/to/input.pdf" \
+  --output-dir "/absolute/path/to/results" \
+  --pages "1,3,8-12"
+```
+
+Only selected pages are rendered, detected, analyzed, and written to `output.md`. Docling receives the smallest continuous physical-page range covering the selection, then its output is filtered to the exact selected pages. For example, `8,9` processes only pages 8–9 in Docling, while `1,3,8-12` processes pages 1–12 in Docling and retains only the requested pages downstream. Metadata records both the complete PDF page count and `selected_pages`; the review comparison displays only those physical pages.
+
+During conversion, concise progress is written to standard error: pipeline stages, rendered-page counts, detection batches, and completed table/image Agent tasks. The final machine-readable result remains the only content written to standard output.
+
+Docling runs in an isolated child process. If that process fails during PDF parsing, the converter keeps the main pipeline alive and retries that PDF once with one Docling parsing thread and batch sizes of one. This fallback is part of the core converter, so it applies equally to `convert`, `run-all`, SDK-driven conversions, and directory batches. User-supplied Docling options, including OCR settings, are retained during the fallback. A second failure is reported normally and can be resumed by running the same command again.
+
+All Agent work uses one dynamic scheduler with at most five workers. Whenever a worker becomes free, it selects the first ready task type in this order: Find, cross-page table, ordinary table, image. Completed Find batches can release bounded table groups before the remaining Find batches finish. Tasks with unresolved neighboring-page dependencies remain blocked rather than being extracted prematurely.
+
+## `convert` parameters
+
+Syntax:
+
+```text
+llmpdf convert PDF --output-dir OUTPUT_DIR [OPTIONS]
+```
+
+### Input and output
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `PDF` | Required | Input PDF path. |
+| `--output-dir PATH` | Required | Root directory for all batch results. |
+| `--batch-id NAME` | UTC timestamp | Batch directory name. It must be a single path component and remain unchanged for retries. |
+| `--pages PAGES` | All pages | One-based PDF pages, such as `1-25` or `1,3,8-12`. Pages are de-duplicated and processed in source order. |
+| `--force` | Disabled | Ignore existing results and run the conversion again. |
+| `--retain-docling-tables` / `--no-retain-docling-tables` | Enabled | Preserve original Docling table Markdown for review. |
+
+### Table and page detection
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `--model MODEL` | `gpt-5.6-sol` | Model used for table detection and extraction. |
+| `--thinking LEVEL` | `medium` | Reasoning effort for table operations, such as `low`, `medium`, or `high`. |
+| `--detection-dpi N` | `96` | Page-rendering DPI for detection. Higher values improve detail but cost more time and memory. |
+| `--contact-sheet-size N` | `8` | Number of PDF pages per detection contact sheet. |
+| `--agent-concurrency N` | `5` | Maximum concurrent Agent tasks across all task types. Must be between `1` and `5`. |
+| `--find-concurrency N` | `5` | Maximum concurrent Find Agent tasks within the global limit. |
+| `--confidence-threshold N` | `0.35` | Minimum candidate confidence, from `0` to `1`. |
+| `--table-concurrency N` | `5` | Table-processing concurrency. Must be greater than `0`. |
+| `--table-image-max-patches N` | `30000` | Maximum 32x32-pixel patches per table Agent image. Oversized table pages lower only their own render DPI. |
+
+### Images and charts
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `--analyze-images` / `--no-analyze-images` | Enabled | Analyze meaningful images and charts. |
+| `--image-model MODEL` | `gpt-5.6-terra` | Model used for image analysis. |
+| `--image-thinking LEVEL` | `medium` | Reasoning effort for image analysis. |
+| `--image-render-dpi N` | `240` | Page-rendering DPI for image analysis. |
+| `--image-max-patches N` | `10000` | Maximum 32x32-pixel patches per image-analysis model image. Oversized pages automatically use a lower DPI; normal pages keep `--image-render-dpi`. |
+| `--image-concurrency N` | `5` | Image-processing concurrency. Must be greater than `0`. |
+
+When an image is a readable chart, the tool attempts to produce structured table data. Visually estimated results are prefixed with “AI visual extraction; values may be inaccurate”. If values cannot be read reliably, the result keeps the image description and does not invent table data.
+
+### Runtime and advanced options
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `--agent-timeout-seconds N` | `1800` | Timeout for each model call, in seconds. |
+| `--pdftoppm PATH` | `pdftoppm` | Command name or executable path for `pdftoppm`. |
+| `--pi-executable PATH` | Auto-detected | Explicit Pi executable path. Normally unnecessary. |
+| `--docling-options-file PATH` | None | JSON file containing additional document-analysis options. |
+| `--keep-sessions` | Disabled | Keep model session files for troubleshooting. |
+| `--keep-work` | Disabled | Keep all intermediate files after success. By default, reproducible work files are removed after metrics and validation are complete. |
+| `--llmpdf-table-executable PATH` | None | Deprecated compatibility option; avoid it in new integrations. |
+
+Use the built-in help as the authoritative parameter reference for the installed version:
+
+```bash
+uv run llmpdf convert --help
+```
+
+## Common commands
+
+### Convert a directory of PDFs
+
+Use `convert-dir` to recursively find PDFs and convert them in place:
+
+```bash
+uv run llmpdf convert-dir "/absolute/path/to/pdf-root" --jobs 3
+```
+
+Each PDF remains beside its generated `output.md`, `assets/`, and `work/` directories. Because these names are shared, in-place mode requires exactly one PDF in each containing directory. Directories containing multiple PDFs are reported as invalid instead of overwriting one result with another.
+
+The directory command skips a PDF only when all of the following are true: `work/status.json` is completed, validation passed, `output.md` is non-empty, and the source SHA-256 in `assets/metadata.json` still matches the PDF. Interrupted, failed, stale, or incomplete results are resumed using valid stage caches.
+
+All PDF workers start immediately. To process up to three PDFs concurrently, run:
+
+```bash
+uv run llmpdf convert-dir "/absolute/path/to/pdf-root" \
+  --jobs 3
+```
+
+When a running PDF finishes, its slot starts the next queued PDF immediately. Status heartbeats are printed every 30 seconds. Combined output is appended to `<pdf-root>/log.md` and synchronized to disk every 300 lines; use `--log-file PATH` to choose another file.
+
+Preview the decisions without converting anything:
+
+```bash
+uv run llmpdf convert-dir "/absolute/path/to/pdf-root" --dry-run
+```
+
+Options for each individual conversion can be forwarded after `--`:
+
+```bash
+uv run llmpdf convert-dir "/absolute/path/to/pdf-root" \
+  --jobs 2 -- --no-analyze-images
+```
+
+Use `--no-recursive` to inspect only PDFs directly inside the specified directory. The process exits nonzero if any PDF fails or any directory contains multiple PDFs.
+
+The final directory layout is:
+
+```text
+<pdf-root>/
+├── log.md
+└── company-or-document/
+    ├── source.pdf
+    ├── output.md
+    ├── assets/
+    └── work/
+```
+
+### Other examples
+
+Process text and tables without analyzing images:
+
+```bash
+uv run llmpdf convert input.pdf \
+  --output-dir results \
+  --batch-id text-and-tables \
+  --no-analyze-images
+```
+
+Reduce concurrency on a resource-constrained machine:
+
+```bash
+uv run llmpdf convert input.pdf \
+  --output-dir results \
+  --batch-id low-load \
+  --agent-concurrency 2 \
+  --find-concurrency 2 \
+  --table-concurrency 2 \
+  --image-concurrency 2
+```
+
+For a scanned document, create `docling-options.json` to enable OCR:
+
+```json
+{
+  "do_ocr": true,
+  "images_scale": 2.0,
+  "ocr_options": {
+    "lang": ["de", "en"]
+  }
+}
+```
+
+Then run:
+
+```bash
+uv run llmpdf convert scanned.pdf \
+  --output-dir results \
+  --batch-id scanned-document \
+  --docling-options-file docling-options.json
+```
+
+Line breaks inside table cells are converted to spaces. Final Markdown does not use `<br>` for cell-internal line breaks.
+
+## Output files
+
+A successful conversion produces:
+
+```text
+<output-dir>/<batch-id>/<pdf-stem>/
+├── output.md
+├── assets/
+│   ├── metadata.json
+│   ├── tables/
+│   ├── images/
+│   ├── chart-tables/
+│   └── docling-tables/
+└── work/
+    ├── status.json
+    ├── metrics.json
+    ├── run-manifest.json
+    ├── run-blocks.json.gz
+    ├── table-code/             # Agent-authored table extractors
+    ├── diagnostics/
+    └── review/                 # Created only after review activity
+```
+
+- `output.md`: final Markdown.
+- `assets/tables/`: extracted table CSV files.
+- `assets/images/`: images retained from the PDF.
+- `assets/chart-tables/`: CSV files extracted from charts.
+- `assets/metadata.json`: page, source, and artifact relationships.
+- `work/status.json`: live and final run state, elapsed time, pipeline and Agent task counts, token usage, and Pi's API-price estimate.
+- `work/metrics.json`: elapsed time, token usage, and status for the full run and individual model calls.
+- `work/run-manifest.json`: source, model, and candidate-page information required to regenerate tables.
+- `work/run-blocks.json.gz`: compressed document structure used when regenerated tables are merged back into Markdown.
+- `work/table-code/`: the generated `extract.py` for every table plus a compact page-mapping index.
+- `work/diagnostics/`: troubleshooting information.
+
+### Run status
+
+`work/status.json` shows whether a conversion is `running`, `completed`, or `failed`. It is updated during the run and retained after cleanup. It also records elapsed time, successful and failed task counts, token usage, and Pi's estimated API cost.
+
+Example:
+
+```json
+{
+  "status": "completed",
+  "elapsed_seconds": 130.0,
+  "pipeline": {
+    "successful_tasks": 11,
+    "failed_tasks": 0
+  },
+  "agents": {
+    "successful_tasks": 12,
+    "failed_tasks": 0
+  },
+  "usage": {
+    "total_tokens": 1700,
+    "pi_api_price_estimate_usd": 0.012345
+  }
+}
+```
+
+`pi_api_price_estimate_usd` is an estimate for comparing runs, not the actual charge against a Codex plan. See `work/metrics.json` for detailed timing and token information.
+
+Successful conversions use minimal retention by default. Raw Agent logs, page renders, single-page PDFs, Docling work files, and duplicate table/image intermediates are removed only after validation passes and metrics are written. Failed conversions keep their work files so the same batch can resume. Use `--keep-work` for a successful diagnostic run that must retain all intermediates. `--keep-sessions` also prevents minimal cleanup.
+
+Artifact paths in `output.md` and metadata are POSIX paths relative to the current result directory, so the entire directory can be moved. Keep `output.md`, `assets/`, and `work/` together. The source PDF must remain available at the path recorded in `assets/metadata.json` for review and table regeneration.
+
+Large images created from full-page vision analysis are stored as WebP at their original pixel dimensions when that representation is smaller. This changes encoding, not resolution.
+
+## Regenerate tables
+
+Regenerate table extraction from a successful minimal result:
+
+```bash
+uv run llmpdf rerun-tables \
+  "/absolute/path/to/results/edis-2024/input"
+```
+
+The command verifies the source PDF against the saved SHA-256, reruns all table candidates, rebuilds table assets and `output.md`, validates the result, updates metrics, and restores minimal retention. Image analysis and table detection are not rerun. Table regeneration is rejected after review decisions have been saved, preventing accidental loss of review work.
+
+## Manual review
+
+Start the local review UI:
+
+```bash
+uv run --extra review llmpdf review \
+  --result "/absolute/path/to/results/edis-2024/input"
+```
+
+The UI opens at `http://127.0.0.1:8765/` by default. Its document-comparison view displays the PDF on the left and generated Markdown on the right, with synchronized scrolling, a collapsible sidebar, and PDF zoom controls.
+
+Load multiple results by repeating `--result`:
+
+```bash
+uv run --extra review llmpdf review \
+  --result "/path/to/result-a" \
+  --result "/path/to/result-b"
+```
+
+You can also load a complete batch or a review-project file:
+
+```bash
+uv run --extra review llmpdf review --batch "/path/to/batch"
+uv run --extra review llmpdf review --project review-project.json
+```
+
+### `review` parameters
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `--result PATH` | None | Load a result directory. May be repeated. |
+| `--batch PATH` | None | Load a batch directory. May be repeated. |
+| `--project PATH` | None | Load a review-project JSON file. |
+| `--host HOST` | `127.0.0.1` | Listening address. |
+| `--port N` | `8765` | Listening port. |
+| `--no-open` | Disabled | Start the server without opening a browser. |
+
+The review UI operates on local results and does not make model calls.
+
+## AI and automation usage
+
+For non-interactive calls, always provide absolute input and output paths and a stable `--batch-id`:
+
+```bash
+uv run llmpdf convert \
+  "/data/in/report.pdf" \
+  --output-dir "/data/out" \
+  --batch-id "report-2026-09-03"
+```
+
+Calling programs should follow these conventions:
+
+1. Use the process exit code to determine success or failure.
+2. On success, parse JSON from standard output and read the final Markdown path from `output_markdown`.
+3. On failure, parse JSON from standard error. If it contains `batch_id` or `recovery`, retry with the original parameters.
+4. Do not copy `output.md` without its referenced resources; preserve at least `assets/` with it.
+5. Read `work/metrics.json` for cost reporting or slow-call diagnostics.
+
+## Python SDK
+
+```python
+from pathlib import Path
+
+from llmpdf import ConvertOptions, convert
+
+result = convert(
+    ConvertOptions(
+        pdf=Path("/data/in/report.pdf"),
+        output_root=Path("/data/out"),
+        batch_id="report-2026-09-03",
+        pages="1-25",
+        analyze_images=True,
+        agent_concurrency=5,
+        find_concurrency=5,
+        table_concurrency=5,
+        image_concurrency=5,
+    )
+)
+
+print(result.output_markdown)
+print(result.total_tokens)
+print(result.elapsed_seconds)
+```
+
+Path attributes on the SDK result are absolute `Path` objects that can be accessed directly. Artifact paths in `result.to_dict()` are relative to the result directory for portability. See [SDK.md](SDK.md) for all fields and exception types.
+
+## Troubleshooting
+
+### `pdftoppm` is not found
+
+Install Poppler and verify the command:
+
+```bash
+pdftoppm -v
+```
+
+If the executable is outside `PATH`, pass `--pdftoppm /absolute/path/to/pdftoppm`.
+
+### Model authentication fails
+
+Confirm that the local Codex/Pi login is valid and that network access works. Use `--pi-executable` when a custom executable path is required.
+
+### Conversion times out
+
+Complex pages and high-resolution images may need more time. Increase `--agent-timeout-seconds`, or lower `--agent-concurrency` when local resources are limited.
+
+### Resume after a failure
+
+Run the same command with the same `--batch-id`. A new batch name starts a new conversion.
+
+### Regenerate everything
+
+Keep the same path and add `--force`:
+
+```bash
+uv run llmpdf convert input.pdf \
+  --output-dir results \
+  --batch-id existing-batch \
+  --force
+```
+
+## License
+
+llmPDF is distributed under the [llmPDF Limited Use License 1.0](LICENSE).
+Unmodified copies may be used for any purpose, including commercial use, and
+may be redistributed with other software when the required attribution is
+provided. Modification, adaptation, and derivative works are not permitted.