pymupdf4llm-rs 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. pymupdf4llm_rs-0.4.0/.gitattributes +19 -0
  2. pymupdf4llm_rs-0.4.0/.github/ISSUE_TEMPLATE/bug_report.yml +48 -0
  3. pymupdf4llm_rs-0.4.0/.github/ISSUE_TEMPLATE/config.yml +5 -0
  4. pymupdf4llm_rs-0.4.0/.github/ISSUE_TEMPLATE/feature_request.yml +24 -0
  5. pymupdf4llm_rs-0.4.0/.github/PULL_REQUEST_TEMPLATE.md +19 -0
  6. pymupdf4llm_rs-0.4.0/.github/workflows/ci.yml +72 -0
  7. pymupdf4llm_rs-0.4.0/.github/workflows/release-wheels.yml +88 -0
  8. pymupdf4llm_rs-0.4.0/.gitignore +33 -0
  9. pymupdf4llm_rs-0.4.0/CHANGELOG.md +54 -0
  10. pymupdf4llm_rs-0.4.0/CONTRIBUTING.md +46 -0
  11. pymupdf4llm_rs-0.4.0/Cargo.lock +902 -0
  12. pymupdf4llm_rs-0.4.0/Cargo.toml +61 -0
  13. pymupdf4llm_rs-0.4.0/LICENSE +188 -0
  14. pymupdf4llm_rs-0.4.0/PKG-INFO +202 -0
  15. pymupdf4llm_rs-0.4.0/README.md +173 -0
  16. pymupdf4llm_rs-0.4.0/README.zh-CN.md +173 -0
  17. pymupdf4llm_rs-0.4.0/SECURITY.md +34 -0
  18. pymupdf4llm_rs-0.4.0/pyproject.toml +56 -0
  19. pymupdf4llm_rs-0.4.0/python/pymupdf4llm_rs/__init__.py +67 -0
  20. pymupdf4llm_rs-0.4.0/python/pymupdf4llm_rs/__init__.pyi +174 -0
  21. pymupdf4llm_rs-0.4.0/python/pymupdf4llm_rs/py.typed +0 -0
  22. pymupdf4llm_rs-0.4.0/src/async_engine.rs +345 -0
  23. pymupdf4llm_rs-0.4.0/src/bin/pymupdf4llm.rs +138 -0
  24. pymupdf4llm_rs-0.4.0/src/config.rs +548 -0
  25. pymupdf4llm_rs-0.4.0/src/error.rs +36 -0
  26. pymupdf4llm_rs-0.4.0/src/layout/columns.rs +231 -0
  27. pymupdf4llm_rs-0.4.0/src/layout/headers.rs +168 -0
  28. pymupdf4llm_rs-0.4.0/src/layout/mod.rs +198 -0
  29. pymupdf4llm_rs-0.4.0/src/layout/spatial.rs +136 -0
  30. pymupdf4llm_rs-0.4.0/src/layout/tables.rs +448 -0
  31. pymupdf4llm_rs-0.4.0/src/lib.rs +317 -0
  32. pymupdf4llm_rs-0.4.0/src/llama_reader.rs +69 -0
  33. pymupdf4llm_rs-0.4.0/src/markdown/chunk.rs +64 -0
  34. pymupdf4llm_rs-0.4.0/src/markdown/mod.rs +18 -0
  35. pymupdf4llm_rs-0.4.0/src/markdown/renderer.rs +512 -0
  36. pymupdf4llm_rs-0.4.0/src/pdf/content_stream.rs +1236 -0
  37. pymupdf4llm_rs-0.4.0/src/pdf/encoding.rs +1020 -0
  38. pymupdf4llm_rs-0.4.0/src/pdf/font.rs +465 -0
  39. pymupdf4llm_rs-0.4.0/src/pdf/mod.rs +152 -0
  40. pymupdf4llm_rs-0.4.0/src/pdf/objects.rs +136 -0
  41. pymupdf4llm_rs-0.4.0/src/pdf/parser.rs +1248 -0
  42. pymupdf4llm_rs-0.4.0/src/python.rs +1262 -0
  43. pymupdf4llm_rs-0.4.0/src/text_cleaner.rs +155 -0
  44. pymupdf4llm_rs-0.4.0/tests/compatibility_tests.rs +594 -0
  45. pymupdf4llm_rs-0.4.0/tests/test_python_api.py +162 -0
@@ -0,0 +1,19 @@
1
+ # Normalize text line endings to LF
2
+ * text=auto eol=lf
3
+
4
+ # Rust Sources & Configs
5
+ *.rs text eol=lf
6
+ *.toml text eol=lf
7
+ *.lock text eol=lf
8
+ *.md text eol=lf
9
+ *.yml text eol=lf
10
+ *.yaml text eol=lf
11
+ *.sh text eol=lf
12
+
13
+ # Binary Formats
14
+ *.pdf binary
15
+ *.png binary
16
+ *.jpg binary
17
+ *.jpeg binary
18
+ *.webp binary
19
+ *.gz binary
@@ -0,0 +1,48 @@
1
+ name: Bug Report / 缺陷反馈
2
+ description: Report a PDF parsing, Markdown formatting, or mojibake hygiene issue / 提交 PDF 解析、排版或乱码清洗缺陷
3
+ title: "[Bug]: "
4
+ labels: ["bug", "triage"]
5
+ body:
6
+ - type: markdown
7
+ attributes:
8
+ value: |
9
+ Thank you for reporting an issue! Both **English** and **简体中文** descriptions are welcome.
10
+ 感谢您的反馈!支持使用**英文**或**简体中文**填写以下信息。
11
+
12
+ - type: dropdown
13
+ id: component
14
+ attributes:
15
+ label: Affected Module / 受影响模块
16
+ options:
17
+ - PDF Binary / Stream Parser (`src/pdf`)
18
+ - Layout / Multi-Column / Tables (`src/layout`)
19
+ - Text Hygiene / Mojibake (`src/text_cleaner.rs`)
20
+ - Async Engine / CLI (`src/async_engine.rs`, `src/bin`)
21
+ validations:
22
+ required: true
23
+
24
+ - type: textarea
25
+ id: description
26
+ attributes:
27
+ label: Issue Description / 问题描述
28
+ description: Describe expected PyMuPDF4LLM output vs. actual output / 描述期望的 Markdown 输出与实际差异
29
+ placeholder: e.g. Multi-table page merges two tables, or Type0 CJK characters render as mojibake...
30
+ validations:
31
+ required: true
32
+
33
+ - type: textarea
34
+ id: options
35
+ attributes:
36
+ label: Options & Rust Toolchain / 运行参数与环境
37
+ description: Table strategy, header mode, hygiene flags, rustc version / 表格策略、标题模式、清洗开关与 Rust 版本
38
+ placeholder: |
39
+ - table_strategy: TableStrategy::LinesStrict
40
+ - hdr_info: HeaderMode::IdentifyHeaders
41
+ - hygiene: clean_mojibake=true, dehyphenate=true, dedup_running_headers=false
42
+ - rustc --version:
43
+
44
+ - type: textarea
45
+ id: reproduction
46
+ attributes:
47
+ label: Minimal PDF Sample or Steps / 最小复现步骤或 PDF 特征
48
+ description: Please share PDF producer, font encoding (Type0/TrueType/Type1), or sanitized snippet / 请说明 PDF 生成器、字体编码或脱敏样本
@@ -0,0 +1,5 @@
1
+ blank_issues_enabled: false
2
+ contact_links:
3
+ - name: Security Vulnerability / 安全漏洞私密报告
4
+ url: ../../security/advisories/new
5
+ about: Report PDF parser DoS, decompression bombs, or path traversal privately / 私密反馈解析器崩溃、解压炸弹或路径安全隐患
@@ -0,0 +1,24 @@
1
+ name: Feature or Parity Request / 功能与兼容需求
2
+ description: Request PyMuPDF4LLM API parity, PDF stream filter, or performance optimization / 提交新特性、PDF 编码兼容或性能优化建议
3
+ title: "[Feat]: "
4
+ labels: ["enhancement"]
5
+ body:
6
+ - type: markdown
7
+ attributes:
8
+ value: |
9
+ Both **English** and **简体中文** proposals are welcome.
10
+ 欢迎使用**英文**或**简体中文**提交功能建议与 `pymupdf4llm` 兼容性需求。
11
+
12
+ - type: textarea
13
+ id: summary
14
+ attributes:
15
+ label: Feature Summary / 需求概述
16
+ description: What capability or PyMuPDF4LLM parameter would you like added? / 希望增加什么能力或兼容参数?
17
+ validations:
18
+ required: true
19
+
20
+ - type: textarea
21
+ id: rationale
22
+ attributes:
23
+ label: Use Case & Algorithms / 使用场景与实现思路
24
+ description: Explain the PDF structure, encoding, or concurrency bottleneck / 描述对应 PDF 结构特征、编码规范或并发瓶颈
@@ -0,0 +1,19 @@
1
+ ## Summary / 变更说明
2
+
3
+ <!-- Summarize what changed in English and/or 简体中文 -->
4
+ -
5
+
6
+ ## Type of Change / 变更类型
7
+
8
+ - [ ] PDF Format / Encoding Compatibility (`XRef`, `LZWDecode`, `ToUnicode`, `Form XObject`, etc.)
9
+ - [ ] Text Hygiene & Noise Cleanup (`mojibake`, `3 Tr OCR`, watermarks, de-hyphenation, running headers)
10
+ - [ ] Layout & Algorithms (`SpatialGrid`, `UnionFind`, X-Y Cut, header clustering)
11
+ - [ ] Async & Concurrency / CLI (`Tokio`, `Rayon`, `DashMap`, `pymupdf4llm` CLI)
12
+ - [ ] Documentation / CI (`README`, `CHANGELOG`, `.github`)
13
+
14
+ ## Verification Checklist / 验证清单
15
+
16
+ - [ ] `cargo fmt --all -- --check`
17
+ - [ ] `cargo clippy --all-targets --all-features -- -D warnings`
18
+ - [ ] `cargo test --all-features`
19
+ - [ ] Updated bilingual docs (`README.md` / `README.zh-CN.md` / `CHANGELOG.md`) if public API changed
@@ -0,0 +1,72 @@
1
+ name: CI (Pure-Rust Crate)
2
+
3
+ on:
4
+ push:
5
+ branches: [main, master]
6
+ pull_request:
7
+ branches: [main, master]
8
+
9
+ env:
10
+ CARGO_TERM_COLOR: always
11
+
12
+ jobs:
13
+ rust:
14
+ name: Build, Clippy & Test (pymupdf4llm-rs)
15
+ runs-on: ubuntu-latest
16
+ steps:
17
+ - name: Checkout repository
18
+ uses: actions/checkout@v4
19
+
20
+ - name: Install stable Rust toolchain
21
+ uses: dtolnay/rust-toolchain@stable
22
+ with:
23
+ components: rustfmt, clippy
24
+
25
+ - name: Cache Cargo artifacts
26
+ uses: actions/cache@v4
27
+ with:
28
+ path: |
29
+ ~/.cargo/bin/
30
+ ~/.cargo/registry/index/
31
+ ~/.cargo/registry/cache/
32
+ ~/.cargo/git/db/
33
+ target/
34
+ key: ${{ runner.os }}-cargo-${{ hashFiles('Cargo.toml') }}
35
+
36
+ - name: Check formatting (rustfmt)
37
+ run: cargo fmt --all -- --check
38
+
39
+ - name: Run Clippy lints
40
+ run: cargo clippy --all-targets --features "async,cli,images" -- -D warnings
41
+
42
+ - name: Run unit & compatibility tests
43
+ run: cargo test --features "async,cli,images"
44
+
45
+ - name: Check feature combinations
46
+ run: |
47
+ cargo check --no-default-features
48
+ cargo check --no-default-features --features async
49
+ cargo check --features "async,cli,images"
50
+
51
+ python-pyo3:
52
+ name: Build & Test Python Package (PyO3 + maturin)
53
+ runs-on: ubuntu-latest
54
+ steps:
55
+ - name: Checkout repository
56
+ uses: actions/checkout@v4
57
+
58
+ - name: Set up Python
59
+ uses: actions/setup-python@v5
60
+ with:
61
+ python-version: "3.11"
62
+
63
+ - name: Install stable Rust toolchain
64
+ uses: dtolnay/rust-toolchain@stable
65
+
66
+ - name: Install maturin and pytest
67
+ run: pip install --upgrade pip maturin pytest
68
+
69
+ - name: Build extension module & run pytest
70
+ run: |
71
+ maturin develop --release
72
+ pytest -v tests/test_python_api.py
@@ -0,0 +1,88 @@
1
+ name: Release Python Wheels
2
+
3
+ on:
4
+ push:
5
+ tags:
6
+ - "v*"
7
+ workflow_dispatch:
8
+
9
+ permissions:
10
+ contents: write
11
+
12
+ jobs:
13
+ linux:
14
+ runs-on: ${{ matrix.runner }}
15
+ strategy:
16
+ matrix:
17
+ include:
18
+ - runner: ubuntu-latest
19
+ target: x86_64
20
+ - runner: ubuntu-24.04-arm
21
+ target: aarch64
22
+ steps:
23
+ - uses: actions/checkout@v4
24
+ - uses: actions/setup-python@v5
25
+ with:
26
+ python-version: "3.11"
27
+ - name: Build abi3 wheels (manylinux)
28
+ uses: PyO3/maturin-action@v1
29
+ with:
30
+ target: ${{ matrix.target }}
31
+ args: --release --out dist --find-interpreter
32
+ manylinux: auto
33
+ - uses: actions/upload-artifact@v4
34
+ with:
35
+ name: wheels-linux-${{ matrix.target }}
36
+ path: dist
37
+
38
+ macos:
39
+ runs-on: macos-14
40
+ strategy:
41
+ matrix:
42
+ target: [aarch64-apple-darwin, x86_64-apple-darwin]
43
+ steps:
44
+ - uses: actions/checkout@v4
45
+ - uses: actions/setup-python@v5
46
+ with:
47
+ python-version: "3.11"
48
+ - name: Build abi3 wheels (macOS)
49
+ uses: PyO3/maturin-action@v1
50
+ with:
51
+ target: ${{ matrix.target }}
52
+ args: --release --out dist --find-interpreter
53
+ - uses: actions/upload-artifact@v4
54
+ with:
55
+ name: wheels-macos-${{ matrix.target }}
56
+ path: dist
57
+
58
+ windows:
59
+ runs-on: windows-latest
60
+ steps:
61
+ - uses: actions/checkout@v4
62
+ - uses: actions/setup-python@v5
63
+ with:
64
+ python-version: "3.11"
65
+ - name: Build abi3 wheels (Windows x64)
66
+ uses: PyO3/maturin-action@v1
67
+ with:
68
+ target: x64
69
+ args: --release --out dist --find-interpreter
70
+ - uses: actions/upload-artifact@v4
71
+ with:
72
+ name: wheels-windows-x64
73
+ path: dist
74
+
75
+ release:
76
+ needs: [linux, macos, windows]
77
+ runs-on: ubuntu-latest
78
+ if: startsWith(github.ref, 'refs/tags/v')
79
+ steps:
80
+ - uses: actions/download-artifact@v4
81
+ with:
82
+ pattern: wheels-*
83
+ path: dist
84
+ merge-multiple: true
85
+ - name: Publish wheels to GitHub Release
86
+ uses: softprops/action-gh-release@v2
87
+ with:
88
+ files: dist/*.whl
@@ -0,0 +1,33 @@
1
+ # Rust / Cargo
2
+ /target/
3
+ **/*.rs.bk
4
+ *.pdb
5
+
6
+ # Python / Maturin
7
+ *.so
8
+ *.pyd
9
+ *.whl
10
+ __pycache__/
11
+ .pytest_cache/
12
+ .venv/
13
+ *.egg-info/
14
+ python/pymupdf4llm_rs/*.so
15
+ python/pymupdf4llm_rs/*.pyd
16
+
17
+ # AI Studio Runtime Bridge (not required when consuming Rust crate)
18
+ node_modules/
19
+ dist/
20
+ .playwright-cli/
21
+
22
+ # Environment
23
+ .env
24
+ .env.*
25
+
26
+ # Editor & OS Artifacts
27
+ .DS_Store
28
+ Thumbs.db
29
+ *.log
30
+ .vscode/*
31
+ !.vscode/extensions.json
32
+ .idea/
33
+ .aistudio/
@@ -0,0 +1,54 @@
1
+ # Changelog / 更新日志
2
+
3
+ All notable changes to `pymupdf4llm-rs` are documented in this file.
4
+ 本文档记录 `pymupdf4llm-rs` 纯 Rust 核心库的重要版本变更。
5
+
6
+ ---
7
+
8
+ ## [0.4.0] - 2026-11-16
9
+
10
+ ### English
11
+ - **PDF Format & Encoding Compatibility**:
12
+ - Added `/Pages` tree inheritance for `/MediaBox`, `/CropBox`, `/Rotate`, and `/Resources`.
13
+ - Added `/Subtype /Form` XObject recursive stream execution with `/Matrix` & font stack isolation.
14
+ - Added pure-Rust `LZWDecode` decompression (9–12 bit variable codes with `EarlyChange`).
15
+ - Added `Type0`/CID `/W` width parsing, array-form `beginbfrange` `ToUnicode` CMap decoding, `MacRomanEncoding`, and 70+ Adobe Glyph names.
16
+ - Added `/Rotate` (0°/90°/180°/270°) page coordinate normalization and `DashMap` lock-free font caching.
17
+ - **Text Hygiene & Noise Cleanup**:
18
+ - Added `TextHygieneConfig` & `HygieneStats` diagnostics (`clean_mojibake`, `dehyphenate`, `dedup_faux_bold`, `ignore_invisible_text`, `strip_watermarks`, `dedup_running_headers`).
19
+ - Stripped `3 Tr` invisible OCR overlays, diagonal/low-alpha watermarks, unmapped PUA/control mojibake, and faux-bold overprint duplicates.
20
+ - Added ligature expansion (`ff`, `fi`, `fl`, `ffi`, `ffl`, `ſt`), PUA bullet normalization, line-wrap de-hyphenation, and adjacent same-URI hyperlink merging.
21
+ - **Algorithms & Performance**:
22
+ - Added 2D `SpatialGrid` for $O(1)$ average-case link attachment and table cell span assignment.
23
+ - Added `UnionFind` (DSU) connected components to isolate multiple independent tables per page and cluster vector graphics.
24
+ - **Repository Structure & Python Package (`PyO3` + `maturin`)**:
25
+ - Promoted `pymupdf4llm-rs` to repository root as a standalone pure-Rust library + CLI crate + Python wheel (`abi3-py39`).
26
+ - Added reusable `PdfExtractor` and parse-once `ParsedPdf` Python classes holding shared `Arc<PdfEngine>`, `Arc<ToMarkdownOptions>`, Tokio runtime, global `ToUnicode` CMap cache, and `max_pdf_bytes` / `max_pages` safety guards with GIL release (`py.allow_threads`).
27
+ - Enforced 64 MB decompression-bomb limits across `FlateDecode`, `LZWDecode`, `ASCII85Decode`, and `RunLengthDecode`.
28
+
29
+ ### 简体中文
30
+ - **PDF 格式与文字编码兼容性扩展**:
31
+ - 支持 `/Pages` 树向子页面继承 `/MediaBox`、`/CropBox`、`/Rotate` 与 `/Resources`。
32
+ - 支持 `/Subtype /Form` 表单 XObject 递归解释执行及局部变换矩阵与字体栈叠加。
33
+ - 实现纯 Rust 版 `LZWDecode` 解压器(支持 9–12 位可变码长与 `EarlyChange`)。
34
+ - 完善 `Type0`/CID `/W` 宽度解析、数组型 `beginbfrange` `ToUnicode` CMap、`MacRomanEncoding` 与 70+ Adobe Glyph 字符名映射。
35
+ - 增加 `/Rotate`(0°/90°/180°/270°)页面旋转坐标归一化与无锁 `DashMap` 字体缓存。
36
+ - **乱码治理与噪声过滤**:
37
+ - 新增 `TextHygieneConfig` 与 `HygieneStats` 诊断统计(`clean_mojibake`、`dehyphenate`、`dedup_faux_bold`、`ignore_invisible_text`、`strip_watermarks`、`dedup_running_headers`)。
38
+ - 自动过滤 `3 Tr` 隐形 OCR 层、斜向/低透明度水印、不可映射 PUA/控制符乱码以及伪粗体(Faux-Bold)重影。
39
+ - 支持连字展开、PUA 项目符号归一化、换行断词拼接及相邻同 URI 超链接 Span 合并。
40
+ - **空间算法与性能优化**:
41
+ - 引入二维 `SpatialGrid` 空间网格索引,将链接命中与表格单元匹配降至均摊 $O(1)$。
42
+ - 引入并查集 `UnionFind` 连通分量算法,精准分离单页多张独立表格并高效聚类矢量图形。
43
+ - **仓库结构精简**:
44
+ - 移除前端展示代码,将纯 Rust 工具包与 CLI 主程序置于仓库根目录。
45
+
46
+ ---
47
+
48
+ ## [0.1.0] - 2026-11-16
49
+
50
+ ### English
51
+ - Initial pure-Rust `pymupdf4llm-rs` crate with PDF 1.0–2.0 parser, `FlateDecode`, recursive X-Y Cut multi-column ordering, `IdentifyHeaders` / `TocHeaders`, 4 table strategies, Tokio + Rayon concurrency, and CLI binary.
52
+
53
+ ### 简体中文
54
+ - 初始版本发布:100% 纯 Rust `pymupdf4llm-rs` 核心库,支持 PDF 1.0–2.0 解析、`FlateDecode`、递归 X-Y Cut 多栏切分、`IdentifyHeaders`/`TocHeaders`、4 种表格策略、Tokio + Rayon 并发调度与 CLI 工具。
@@ -0,0 +1,46 @@
1
+ # Contributing to pymupdf4llm-rs / 参与贡献指南
2
+
3
+ Thank you for contributing to `pymupdf4llm-rs`! Bug reports, PDF compatibility fixtures, performance optimizations, and documentation improvements are welcome in both **English** and **简体中文**.
4
+
5
+ 感谢您关注并参与 `pymupdf4llm-rs`!欢迎提交 Bug 反馈、疑难 PDF 兼容性测试用例、性能优化以及文档完善(支持**英文**或**简体中文**)。
6
+
7
+ ---
8
+
9
+ ## 1. Development Setup / 本地开发环境
10
+
11
+ - Rust `1.78+` (Edition 2021)
12
+ - 100% pure-Rust dependencies only (`no C/C++ FFI`) / 仅允许纯 Rust 依赖,严禁引入 C/C++ FFI
13
+
14
+ ```bash
15
+ cargo fmt --all -- --check
16
+ cargo clippy --all-targets --all-features -- -D warnings
17
+ cargo test --all-features
18
+ ```
19
+
20
+ ---
21
+
22
+ ## 2. Coding Guidelines / 开发规范
23
+
24
+ 1. **Zero C/C++ FFI / 零外部原生依赖**:Keep all parsing, decompression, and layout code in safe pure Rust.
25
+ 2. **Algorithmic Complexity / 算法复杂度控制**:Use `SpatialGrid` and `UnionFind` to avoid $O(N^2)$ scans on dense PDF pages.
26
+ 3. **Tests / 补充测试**:Add unit or integration tests in `tests/compatibility_tests.rs` for any new PDF syntax, font encoding, or hygiene rule.
27
+
28
+ ---
29
+
30
+ ## 3. Commit Convention / 提交信息规范
31
+
32
+ We follow [Conventional Commits](https://www.conventionalcommits.org/):
33
+
34
+ - `feat(pdf): add RunLengthDecode stream filter / 新增 RunLengthDecode 流解压`
35
+ - `fix(hygiene): preserve compound hyphenated terms across line wraps / 修复跨行复合词断字拼接误伤`
36
+ - `perf(layout): optimize SpatialGrid cell sizing / 优化 SpatialGrid 网格粒度`
37
+ - `docs: update README.zh-CN.md / 更新中文文档`
38
+
39
+ ---
40
+
41
+ ## 4. Pull Request Checklist / PR 自检清单
42
+
43
+ - [ ] `cargo fmt --all -- --check` passes.
44
+ - [ ] `cargo clippy --all-targets --all-features -- -D warnings` passes.
45
+ - [ ] `cargo test --all-features` passes.
46
+ - [ ] Bilingual docs (`README.md` / `README.zh-CN.md` / `CHANGELOG.md`) updated if public API changed.