llm-markdownify 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,28 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.pyo
5
+ *.pyd
6
+ *.so
7
+
8
+ # Envs
9
+ .venv/
10
+ .uv/
11
+ .env
12
+ .env.*
13
+
14
+ # Build
15
+ build/
16
+ dist/
17
+ *.egg-info/
18
+
19
+ # IDE
20
+ .vscode/
21
+ .idea/
22
+
23
+ # Tests
24
+ .coverage
25
+ htmlcov/
26
+ .pytest_cache/
27
+
28
+ test_files/
@@ -0,0 +1,73 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
10
+
11
+ "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
12
+
13
+ "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
14
+
15
+ "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
16
+
17
+ "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
18
+
19
+ "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
20
+
21
+ "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
22
+
23
+ "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
24
+
25
+ "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
26
+
27
+ "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
28
+
29
+ 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
30
+
31
+ 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
32
+
33
+ 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
34
+
35
+ (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
36
+
37
+ (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
38
+
39
+ (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
40
+
41
+ (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
42
+
43
+ You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
44
+
45
+ 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
46
+
47
+ 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
48
+
49
+ 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
50
+
51
+ 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
52
+
53
+ 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
54
+
55
+ END OF TERMS AND CONDITIONS
56
+
57
+ APPENDIX: How to apply the Apache License to your work.
58
+
59
+ To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
60
+
61
+ Copyright [yyyy] [name of copyright owner]
62
+
63
+ Licensed under the Apache License, Version 2.0 (the "License");
64
+ you may not use this file except in compliance with the License.
65
+ You may obtain a copy of the License at
66
+
67
+ http://www.apache.org/licenses/LICENSE-2.0
68
+
69
+ Unless required by applicable law or agreed to in writing, software
70
+ distributed under the License is distributed on an "AS IS" BASIS,
71
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
72
+ See the License for the specific language governing permissions and
73
+ limitations under the License.
@@ -0,0 +1,6 @@
1
+ llm-markdownify - Convert documents to Markdown using Vision LLMs
2
+
3
+ Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
4
+
5
+ This product includes software developed by Sethu Pavan Venkata Reddy Pastula.
6
+ Project: https://github.com/sethupavan12/Markdownify
@@ -0,0 +1,165 @@
1
+ Metadata-Version: 2.4
2
+ Name: llm-markdownify
3
+ Version: 0.1.0
4
+ Summary: Convert documents (PDF, DOCX) to high-quality Markdown using Vision LLMs via LiteLLM
5
+ Project-URL: Homepage, https://github.com/sethupavan12/Markdownify
6
+ Project-URL: Repository, https://github.com/sethupavan12/Markdownify
7
+ Project-URL: Issues, https://github.com/sethupavan12/Markdownify/issues
8
+ Author: Sethu Pavan Venkata Reddy Pastula
9
+ License: Apache-2.0
10
+ License-File: LICENSE
11
+ License-File: NOTICE
12
+ Keywords: docx,litellm,llm,markdown,ocr,pdf,vision
13
+ Classifier: License :: OSI Approved :: Apache Software License
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Topic :: Text Processing :: Markup
17
+ Classifier: Topic :: Utilities
18
+ Requires-Python: >=3.10
19
+ Requires-Dist: litellm[proxy]>=1.43.0
20
+ Requires-Dist: pillow>=10.3.0
21
+ Requires-Dist: pydantic>=2.7.0
22
+ Requires-Dist: pypdfium2>=4.30.0
23
+ Requires-Dist: tqdm>=4.66.0
24
+ Requires-Dist: typer>=0.12.3
25
+ Provides-Extra: dev
26
+ Requires-Dist: pre-commit>=3.6.0; extra == 'dev'
27
+ Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
28
+ Requires-Dist: pytest>=8.2.0; extra == 'dev'
29
+ Requires-Dist: ruff>=0.5.6; extra == 'dev'
30
+ Provides-Extra: docx
31
+ Requires-Dist: docx2pdf>=0.1.8; extra == 'docx'
32
+ Description-Content-Type: text/markdown
33
+
34
+ ## Markdownify
35
+
36
+ Mardownify is a super easy-to-use PDF to high-quality Markdown converter using Vision LLMs. It supports text, images, signatures, tables, charts, flowcharts and preserves document structure (Headings, numbered lists etc).
37
+
38
+ Tables become Markdown tables, charts become Mermaid diagrams, and images get concise summaries. Use as a CLI or Python library. Works with 100+ LLMs. Recommended to use with `gpt-5-mini` or `gpt-4.1-mini` or even better models for better performance.
39
+
40
+ ### Features
41
+ - High-quality complex markdown generation powered by LLMs.
42
+ - Supports Text, Images, Tables, Charts.
43
+ - Built-in prompts tuned for clean Markdown, Mermaid, and structured headings along with ability to customise.
44
+ - Supports multi-page tables, charts and images.
45
+ - High-fidelity page rendering from PDF.
46
+ - Optional DOCX→PDF conversion using MS word installation.
47
+ - Works seamlessly with 100+ LLMs with LiteLLM Intergration.
48
+
49
+ ### Install
50
+ ```bash
51
+ uv pip install llm-markdownify
52
+ # or
53
+ pip install llm-markdownify
54
+ ```
55
+
56
+ ### Quickstart (CLI)
57
+ ```bash
58
+ markdownify run input.pdf -o output.md --model gpt-5-mini
59
+ ```
60
+
61
+ ### Python API (one-liner)
62
+ ```python
63
+ from llm_markdownify import convert
64
+
65
+ convert(
66
+ "input.pdf",
67
+ "output.md",
68
+ model="gpt-5-mini", # optional; can rely on env/provider defaults
69
+ dpi=72,
70
+ profile="contracts", # or path to JSON profile
71
+ )
72
+ ```
73
+
74
+ Optional DOCX support (macOS/Windows via Word):
75
+ ```bash
76
+ pip install llm-markdownify[docx]
77
+ ```
78
+
79
+ ### Configure your provider (via LiteLLM)
80
+ Pick one of the following. See the full providers list and details in the LiteLLM docs: [Supported Providers](https://docs.litellm.ai/docs/providers).
81
+
82
+ - **OpenAI**
83
+ - Set your API key:
84
+ ```bash
85
+ export OPENAI_API_KEY="sk-..."
86
+ ```
87
+ - Example usage:
88
+ ```bash
89
+ markdownify run input.pdf -o output.md --model gpt-5-mini
90
+ ```
91
+
92
+ - **Google Gemini**
93
+ - Set your API key (Google AI Studio key):
94
+ ```bash
95
+ export GOOGLE_API_KEY="..."
96
+ ```
97
+ - Example usage (pick a Gemini vision-capable model):
98
+ ```bash
99
+ markdownify run input.pdf -o output.md --model gemini/gemini-2.5-flash
100
+ ```
101
+
102
+ - **Azure OpenAI**
103
+ - Set these environment variables (values from your Azure OpenAI resource):
104
+ ```bash
105
+ export AZURE_API_KEY="..."
106
+ export AZURE_API_BASE="https://<your-resource>.openai.azure.com"
107
+ export AZURE_API_VERSION=""
108
+ ```
109
+ - Use your deployment name via the `azure/<deployment_name>` model syntax:
110
+ ```bash
111
+ markdownify run input.pdf -o output.md --model azure/<deployment_name>
112
+ ```
113
+ - See: [LiteLLM Azure OpenAI](https://docs.litellm.ai/docs/providers/azure_openai)
114
+
115
+ - **OpenAI-compatible APIs**
116
+ - Many providers expose an OpenAI-compatible REST API. Set your API key and base URL:
117
+ ```bash
118
+ export OPENAI_API_KEY="..."
119
+ export OPENAI_API_BASE="https://your-openai-compatible-endpoint.com/v1"
120
+ ```
121
+ - Use the model name supported by that endpoint:
122
+ ```bash
123
+ markdownify run input.pdf -o output.md --model <model-name>
124
+ ```
125
+ - Reference: [LiteLLM Providers](https://docs.litellm.ai/docs/providers)
126
+
127
+ For additional providers and advanced configuration (fallbacks, cost tracking, streaming), see the LiteLLM docs: [Getting Started](https://docs.litellm.ai/).
128
+
129
+ ### Configuration flags
130
+ - `--model`: LiteLLM model (e.g., `gpt-5-mini`, `azure/<deployment>`, `gemini/gemini-2.5-flash`)
131
+ - `--dpi`: Render DPI (default 72)
132
+ - `--max-group-pages`: Max pages to merge for continued content (default 3)
133
+ - `--no-grouping`: Disable LLM-based grouping
134
+ - `--temperature`, `--max-tokens`: LLM generation params
135
+
136
+ ### Dev tooling
137
+ - Install pre-commit and enable the license header hook:
138
+ ```bash
139
+ pip install pre-commit
140
+ pre-commit install
141
+ # run on all files once
142
+ pre-commit run --all-files
143
+ ```
144
+ This inserts the header:
145
+ ```
146
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
147
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
148
+ # SPDX-License-Identifier: Apache-2.0
149
+ ```
150
+
151
+ ### Attribution & License
152
+ This project uses the Apache 2.0 License, which includes an attribution/NOTICE requirement. If you distribute or use this project, please keep the `LICENSE` and `NOTICE` files intact, crediting the original author, Sethu Pavan Venkata Reddy Pastula.
153
+
154
+ - Project repository: https://github.com/sethupavan12/Markdownify
155
+
156
+ ### Development
157
+ - Requires Python 3.10+
158
+ - Use `uv` for fast installs: `uv sync`
159
+ - Run tests: `pytest`
160
+ - Lint: `ruff check src tests`
161
+
162
+ ### Releasing
163
+ GitHub Actions are configured to:
164
+ - Run tests on PRs/pushes
165
+ - Build & publish to PyPI on tagged releases (requires `PYPI_API_TOKEN` secret)
@@ -0,0 +1,132 @@
1
+ ## Markdownify
2
+
3
+ Mardownify is a super easy-to-use PDF to high-quality Markdown converter using Vision LLMs. It supports text, images, signatures, tables, charts, flowcharts and preserves document structure (Headings, numbered lists etc).
4
+
5
+ Tables become Markdown tables, charts become Mermaid diagrams, and images get concise summaries. Use as a CLI or Python library. Works with 100+ LLMs. Recommended to use with `gpt-5-mini` or `gpt-4.1-mini` or even better models for better performance.
6
+
7
+ ### Features
8
+ - High-quality complex markdown generation powered by LLMs.
9
+ - Supports Text, Images, Tables, Charts.
10
+ - Built-in prompts tuned for clean Markdown, Mermaid, and structured headings along with ability to customise.
11
+ - Supports multi-page tables, charts and images.
12
+ - High-fidelity page rendering from PDF.
13
+ - Optional DOCX→PDF conversion using MS word installation.
14
+ - Works seamlessly with 100+ LLMs with LiteLLM Intergration.
15
+
16
+ ### Install
17
+ ```bash
18
+ uv pip install llm-markdownify
19
+ # or
20
+ pip install llm-markdownify
21
+ ```
22
+
23
+ ### Quickstart (CLI)
24
+ ```bash
25
+ markdownify run input.pdf -o output.md --model gpt-5-mini
26
+ ```
27
+
28
+ ### Python API (one-liner)
29
+ ```python
30
+ from llm_markdownify import convert
31
+
32
+ convert(
33
+ "input.pdf",
34
+ "output.md",
35
+ model="gpt-5-mini", # optional; can rely on env/provider defaults
36
+ dpi=72,
37
+ profile="contracts", # or path to JSON profile
38
+ )
39
+ ```
40
+
41
+ Optional DOCX support (macOS/Windows via Word):
42
+ ```bash
43
+ pip install llm-markdownify[docx]
44
+ ```
45
+
46
+ ### Configure your provider (via LiteLLM)
47
+ Pick one of the following. See the full providers list and details in the LiteLLM docs: [Supported Providers](https://docs.litellm.ai/docs/providers).
48
+
49
+ - **OpenAI**
50
+ - Set your API key:
51
+ ```bash
52
+ export OPENAI_API_KEY="sk-..."
53
+ ```
54
+ - Example usage:
55
+ ```bash
56
+ markdownify run input.pdf -o output.md --model gpt-5-mini
57
+ ```
58
+
59
+ - **Google Gemini**
60
+ - Set your API key (Google AI Studio key):
61
+ ```bash
62
+ export GOOGLE_API_KEY="..."
63
+ ```
64
+ - Example usage (pick a Gemini vision-capable model):
65
+ ```bash
66
+ markdownify run input.pdf -o output.md --model gemini/gemini-2.5-flash
67
+ ```
68
+
69
+ - **Azure OpenAI**
70
+ - Set these environment variables (values from your Azure OpenAI resource):
71
+ ```bash
72
+ export AZURE_API_KEY="..."
73
+ export AZURE_API_BASE="https://<your-resource>.openai.azure.com"
74
+ export AZURE_API_VERSION=""
75
+ ```
76
+ - Use your deployment name via the `azure/<deployment_name>` model syntax:
77
+ ```bash
78
+ markdownify run input.pdf -o output.md --model azure/<deployment_name>
79
+ ```
80
+ - See: [LiteLLM Azure OpenAI](https://docs.litellm.ai/docs/providers/azure_openai)
81
+
82
+ - **OpenAI-compatible APIs**
83
+ - Many providers expose an OpenAI-compatible REST API. Set your API key and base URL:
84
+ ```bash
85
+ export OPENAI_API_KEY="..."
86
+ export OPENAI_API_BASE="https://your-openai-compatible-endpoint.com/v1"
87
+ ```
88
+ - Use the model name supported by that endpoint:
89
+ ```bash
90
+ markdownify run input.pdf -o output.md --model <model-name>
91
+ ```
92
+ - Reference: [LiteLLM Providers](https://docs.litellm.ai/docs/providers)
93
+
94
+ For additional providers and advanced configuration (fallbacks, cost tracking, streaming), see the LiteLLM docs: [Getting Started](https://docs.litellm.ai/).
95
+
96
+ ### Configuration flags
97
+ - `--model`: LiteLLM model (e.g., `gpt-5-mini`, `azure/<deployment>`, `gemini/gemini-2.5-flash`)
98
+ - `--dpi`: Render DPI (default 72)
99
+ - `--max-group-pages`: Max pages to merge for continued content (default 3)
100
+ - `--no-grouping`: Disable LLM-based grouping
101
+ - `--temperature`, `--max-tokens`: LLM generation params
102
+
103
+ ### Dev tooling
104
+ - Install pre-commit and enable the license header hook:
105
+ ```bash
106
+ pip install pre-commit
107
+ pre-commit install
108
+ # run on all files once
109
+ pre-commit run --all-files
110
+ ```
111
+ This inserts the header:
112
+ ```
113
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
114
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
115
+ # SPDX-License-Identifier: Apache-2.0
116
+ ```
117
+
118
+ ### Attribution & License
119
+ This project uses the Apache 2.0 License, which includes an attribution/NOTICE requirement. If you distribute or use this project, please keep the `LICENSE` and `NOTICE` files intact, crediting the original author, Sethu Pavan Venkata Reddy Pastula.
120
+
121
+ - Project repository: https://github.com/sethupavan12/Markdownify
122
+
123
+ ### Development
124
+ - Requires Python 3.10+
125
+ - Use `uv` for fast installs: `uv sync`
126
+ - Run tests: `pytest`
127
+ - Lint: `ruff check src tests`
128
+
129
+ ### Releasing
130
+ GitHub Actions are configured to:
131
+ - Run tests on PRs/pushes
132
+ - Build & publish to PyPI on tagged releases (requires `PYPI_API_TOKEN` secret)
@@ -0,0 +1,82 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ [build-system]
6
+ requires = ["hatchling>=1.18"]
7
+ build-backend = "hatchling.build"
8
+
9
+ [project]
10
+ name = "llm-markdownify"
11
+ version = "0.1.0"
12
+ description = "Convert documents (PDF, DOCX) to high-quality Markdown using Vision LLMs via LiteLLM"
13
+ readme = "README.md"
14
+ requires-python = ">=3.10"
15
+ license = { text = "Apache-2.0" }
16
+ authors = [
17
+ { name = "Sethu Pavan Venkata Reddy Pastula" }
18
+ ]
19
+ keywords = ["markdown", "pdf", "docx", "llm", "vision", "ocr", "litellm"]
20
+ classifiers = [
21
+ "Programming Language :: Python :: 3",
22
+ "License :: OSI Approved :: Apache Software License",
23
+ "Operating System :: OS Independent",
24
+ "Topic :: Text Processing :: Markup",
25
+ "Topic :: Utilities",
26
+ ]
27
+ dependencies = [
28
+ "litellm[proxy]>=1.43.0",
29
+ "pypdfium2>=4.30.0",
30
+ "Pillow>=10.3.0",
31
+ "typer>=0.12.3",
32
+ "pydantic>=2.7.0",
33
+ "tqdm>=4.66.0",
34
+ ]
35
+
36
+ [project.optional-dependencies]
37
+ docx = [
38
+ "docx2pdf>=0.1.8",
39
+ ]
40
+ dev = [
41
+ "pytest>=8.2.0",
42
+ "pytest-cov>=5.0.0",
43
+ "ruff>=0.5.6",
44
+ "pre-commit>=3.6.0",
45
+ ]
46
+
47
+ [dependency-groups]
48
+ dev = [
49
+ "pytest>=8.2.0",
50
+ "pytest-cov>=5.0.0",
51
+ "ruff>=0.5.6",
52
+ "pre-commit>=3.6.0",
53
+ ]
54
+
55
+ [project.urls]
56
+ Homepage = "https://github.com/sethupavan12/Markdownify"
57
+ Repository = "https://github.com/sethupavan12/Markdownify"
58
+ Issues = "https://github.com/sethupavan12/Markdownify/issues"
59
+
60
+ [project.scripts]
61
+ markdownify = "llm_markdownify.cli:app"
62
+
63
+ [tool.uv]
64
+ default-groups = ["dev"]
65
+
66
+ [tool.pytest.ini_options]
67
+ addopts = "-q"
68
+ pythonpath = ["src"]
69
+
70
+ [tool.ruff]
71
+ line-length = 100
72
+
73
+ [tool.hatch.build.targets.wheel]
74
+ packages = ["src/llm_markdownify"]
75
+
76
+ [tool.hatch.build]
77
+ include = [
78
+ "src/llm_markdownify/**",
79
+ "README.md",
80
+ "LICENSE",
81
+ "NOTICE",
82
+ ]
@@ -0,0 +1,15 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ __all__ = [
6
+ "MarkdownifyConfig",
7
+ "Markdownifier",
8
+ "convert",
9
+ ]
10
+
11
+ __version__ = "0.1.0"
12
+
13
+ from .config import MarkdownifyConfig # noqa: E402
14
+ from .markdownifier import Markdownifier # noqa: E402
15
+ from .api import convert # noqa: E402
@@ -0,0 +1,62 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ from pathlib import Path
8
+ from typing import Optional
9
+
10
+ from .config import MarkdownifyConfig
11
+ from .markdownifier import Markdownifier
12
+
13
+
14
+ def convert(
15
+ input_path: str | Path,
16
+ output_path: str | Path,
17
+ *,
18
+ model: Optional[str] = None,
19
+ dpi: int = 200,
20
+ max_group_pages: int = 3,
21
+ enable_grouping: bool = True,
22
+ temperature: float = 0.2,
23
+ max_tokens: Optional[int] = None,
24
+ concurrency: int = 4,
25
+ profile: Optional[str] = None,
26
+ allow_docx: bool = False,
27
+ ) -> Path:
28
+ """Convert a document to Markdown using the configured LLM via LiteLLM.
29
+
30
+ Parameters
31
+ - input_path: PDF path (preferred) or DOCX if `allow_docx=True`
32
+ - output_path: Markdown file destination
33
+ - model: LiteLLM model name (e.g., 'gpt-4.1-mini', 'azure/<deployment>', 'gemini/gemini-2.5-flash')
34
+ - dpi: Render DPI for PDF pages (higher = slower, clearer)
35
+ - max_group_pages: Max pages to merge when a table/chart spans pages
36
+ - enable_grouping: Whether to use LLM to detect cross-page continuations
37
+ - temperature: LLM temperature
38
+ - max_tokens: LLM max tokens; if None, uses config default
39
+ - concurrency: Max parallel LLM calls across page groups
40
+ - profile: Prompt profile name ('contracts', 'generic') or path to a JSON profile
41
+ - allow_docx: Enable DOCX via Word/COM conversion (not recommended; prefer PDFs)
42
+
43
+ Returns
44
+ - Path to the written Markdown file
45
+ """
46
+ cfg_kwargs = dict(
47
+ input_path=Path(input_path),
48
+ output_path=Path(output_path),
49
+ dpi=dpi,
50
+ max_group_pages=max_group_pages,
51
+ enable_grouping=enable_grouping,
52
+ temperature=temperature,
53
+ concurrency=concurrency,
54
+ allow_docx=allow_docx,
55
+ )
56
+ if model is not None:
57
+ cfg_kwargs["model"] = model
58
+ if max_tokens is not None:
59
+ cfg_kwargs["max_tokens"] = max_tokens
60
+
61
+ cfg = MarkdownifyConfig(**cfg_kwargs)
62
+ return Markdownifier(cfg, profile=profile).run()
@@ -0,0 +1,74 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ from pathlib import Path
8
+ from typing import Optional
9
+
10
+ import typer
11
+
12
+ from .config import MarkdownifyConfig
13
+ from .markdownifier import Markdownifier
14
+
15
+ app = typer.Typer(help="Convert documents (PDF) to Markdown using Vision LLMs via LiteLLM.")
16
+
17
+
18
+ @app.command()
19
+ def run(
20
+ input_path: str = typer.Argument(
21
+ ..., help="Path to input .pdf (preferred) or .docx (discouraged)"
22
+ ),
23
+ output: str = typer.Option(..., "-o", "--output", help="Output .md path"),
24
+ model: Optional[str] = typer.Option(
25
+ None, help="LiteLLM model, e.g. gpt-4.1-mini, azure/<deployment>, gemini/gemini-2.5-flash"
26
+ ),
27
+ dpi: int = typer.Option(
28
+ 72,
29
+ help="DPI for rendering PDF pages. Higher DPI may improve OCR accuracy if document is blurry.",
30
+ ),
31
+ max_group_pages: int = typer.Option(3, help="Max pages to merge for continued content"),
32
+ grouping: bool = typer.Option(
33
+ True,
34
+ "--grouping/--no-grouping",
35
+ help="Enable LLM-based grouping of continued content",
36
+ ),
37
+ temperature: float = typer.Option(
38
+ 0.2, help="LLM temperature. Lower makes OCR results more reliable."
39
+ ),
40
+ max_tokens: Optional[int] = typer.Option(
41
+ None, help="LLM max tokens (defaults to config default). Change based on model limitations"
42
+ ),
43
+ concurrency: int = typer.Option(
44
+ 4,
45
+ help="Max concurrent LLM requests for page groups. Higher concurrency means faster processing at the risk of hitting rate limits.",
46
+ ),
47
+ profile: Optional[str] = typer.Option(
48
+ None, help="Prompt profile name (e.g., 'contracts', 'generic') or path to JSON profile"
49
+ ),
50
+ allow_docx: bool = typer.Option(
51
+ False, help="Allow DOCX via Word/COM conversion (not recommended). Prefer PDFs."
52
+ ),
53
+ ):
54
+ cfg_kwargs = dict(
55
+ input_path=Path(input_path),
56
+ output_path=Path(output),
57
+ dpi=dpi,
58
+ max_group_pages=max_group_pages,
59
+ enable_grouping=grouping,
60
+ temperature=temperature,
61
+ concurrency=concurrency,
62
+ allow_docx=allow_docx,
63
+ )
64
+ if model:
65
+ cfg_kwargs["model"] = model
66
+ if max_tokens is not None:
67
+ cfg_kwargs["max_tokens"] = max_tokens
68
+
69
+ cfg = MarkdownifyConfig(**cfg_kwargs)
70
+ Markdownifier(cfg, profile=profile).run()
71
+
72
+
73
+ if __name__ == "__main__": # pragma: no cover
74
+ app()
@@ -0,0 +1,72 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ import os
8
+ from pathlib import Path
9
+ from typing import Optional
10
+
11
+ from pydantic import BaseModel, Field, field_validator, model_validator
12
+
13
+
14
+ class MarkdownifyConfig(BaseModel):
15
+ """Configuration for the markdownification process."""
16
+
17
+ input_path: Path = Field(..., description="Path to input PDF/DOCX file")
18
+ output_path: Path = Field(..., description="Path to output Markdown file")
19
+
20
+ dpi: int = Field(72, ge=72, le=600, description="DPI used to render PDF pages")
21
+ max_group_pages: int = Field(3, ge=1, le=10, description="Max pages to group together")
22
+ enable_grouping: bool = Field(True, description="Enable LLM-based grouping")
23
+
24
+ # Prefer PDFs; DOCX allowed only with explicit opt-in
25
+ allow_docx: bool = Field(
26
+ False,
27
+ description="Allow DOCX via Word/COM conversion (not recommended). Prefer PDFs.",
28
+ )
29
+
30
+ model: str = Field(
31
+ default_factory=lambda: os.getenv("LLM_MARKDOWNIFY_MODEL", "gpt-4.1-mini"),
32
+ description="LiteLLM model name (e.g., gpt-4.1-mini, azure/<deployment>, gemini/gemini-2.5-flash)",
33
+ )
34
+ temperature: float = Field(0.1, ge=0.0, le=1)
35
+ max_tokens: int = Field(16000, ge=256, le=128000)
36
+
37
+ concurrency: int = Field(
38
+ 4,
39
+ ge=1,
40
+ le=1000,
41
+ description="Max concurrent LLM requests when processing page groups",
42
+ )
43
+
44
+ # Optional path where page images are cached for debugging
45
+ cache_dir: Optional[Path] = Field(None)
46
+
47
+ @field_validator("input_path")
48
+ @classmethod
49
+ def _validate_input(cls, path: Path) -> Path:
50
+ if not path.exists():
51
+ raise ValueError(f"Input file not found: {path}")
52
+ if path.suffix.lower() not in {".pdf", ".docx"}:
53
+ raise ValueError("input_path must be a .pdf or .docx file")
54
+ return path
55
+
56
+ @field_validator("output_path")
57
+ @classmethod
58
+ def _validate_output(cls, path: Path) -> Path:
59
+ parent = path.parent
60
+ if not parent.exists():
61
+ parent.mkdir(parents=True, exist_ok=True)
62
+ if path.suffix.lower() not in {".md", ".markdown"}:
63
+ raise ValueError("output_path must be a .md or .markdown file")
64
+ return path
65
+
66
+ @model_validator(mode="after")
67
+ def _enforce_pdf_preference(self) -> "MarkdownifyConfig":
68
+ if self.input_path.suffix.lower() == ".docx" and not self.allow_docx:
69
+ raise ValueError(
70
+ "DOCX input is not enabled. Prefer exporting to PDF, or rerun with --allow-docx (requires Word/COM)."
71
+ )
72
+ return self
@@ -0,0 +1,52 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ from typing import List
8
+
9
+ from .llm import assess_continuation
10
+ from .pager import PageImage
11
+ from .logging import get_logger
12
+ from .prompt_profiles import PromptProfile
13
+
14
+ logger = get_logger("llm_markdownify.grouping")
15
+
16
+
17
+ def group_pages(
18
+ pages: List[PageImage],
19
+ model: str,
20
+ max_group_pages: int,
21
+ enable_grouping: bool,
22
+ profile: PromptProfile,
23
+ ) -> List[List[PageImage]]:
24
+ if not pages:
25
+ return []
26
+
27
+ if not enable_grouping:
28
+ return [[p] for p in pages]
29
+
30
+ groups: List[List[PageImage]] = []
31
+ current_group: List[PageImage] = [pages[0]]
32
+
33
+ for i in range(len(pages)):
34
+ if i == len(pages) - 1:
35
+ groups.append(current_group)
36
+ break
37
+ a = pages[i]
38
+ b = pages[i + 1]
39
+
40
+ label = assess_continuation(
41
+ model=model, first_data_url=a.data_url, second_data_url=b.data_url, profile=profile
42
+ )
43
+ logger.info("Continuation assessment for pages %d->%d: %s", a.index + 1, b.index + 1, label)
44
+
45
+ continues = label == "CONTINUE_NEXT"
46
+ if continues and len(current_group) < max_group_pages:
47
+ current_group.append(b)
48
+ else:
49
+ groups.append(current_group)
50
+ current_group = [b]
51
+
52
+ return groups
@@ -0,0 +1,54 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ from typing import List
8
+
9
+ import litellm # type: ignore
10
+ from litellm import completion # type: ignore
11
+
12
+ from .prompt_profiles import PromptProfile
13
+
14
+ # Drop unsupported params for strict models (e.g., gpt-5-mini)
15
+ litellm.drop_params = True # type: ignore[attr-defined]
16
+
17
+
18
+ def _message_with_images(text: str, image_data_urls: List[str]) -> dict:
19
+ content = [{"type": "text", "text": text}]
20
+ for url in image_data_urls:
21
+ content.append({"type": "image_url", "image_url": {"url": url}})
22
+ return {"role": "user", "content": content}
23
+
24
+
25
+ def assess_continuation(
26
+ model: str,
27
+ first_data_url: str,
28
+ second_data_url: str | None,
29
+ profile: PromptProfile,
30
+ ) -> str:
31
+ images = [first_data_url] + ([second_data_url] if second_data_url else [])
32
+ messages = [
33
+ {"role": "system", "content": profile.continuation_system},
34
+ _message_with_images(profile.continuation_user, images),
35
+ ]
36
+ resp = completion(model=model, messages=messages, temperature=0.0, max_tokens=4)
37
+ return str(resp["choices"][0]["message"]["content"]).strip().upper()
38
+
39
+
40
+ def generate_markdown(
41
+ model: str,
42
+ image_data_urls: List[str],
43
+ profile: PromptProfile,
44
+ temperature: float = 0.2,
45
+ max_tokens: int = 2000,
46
+ ) -> str:
47
+ messages = [
48
+ {"role": "system", "content": profile.markdown_system},
49
+ _message_with_images(profile.markdown_user, image_data_urls),
50
+ ]
51
+ resp = completion(
52
+ model=model, messages=messages, temperature=temperature, max_tokens=max_tokens
53
+ )
54
+ return str(resp["choices"][0]["message"]["content"]).strip()
@@ -0,0 +1,23 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ import logging
6
+ import sys
7
+
8
+
9
+ def get_logger(name: str) -> logging.Logger:
10
+ logger = logging.getLogger(name)
11
+ if logger.handlers:
12
+ return logger
13
+
14
+ logger.setLevel(logging.INFO)
15
+ handler = logging.StreamHandler(sys.stdout)
16
+ formatter = logging.Formatter(
17
+ fmt="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
18
+ datefmt="%Y-%m-%d %H:%M:%S",
19
+ )
20
+ handler.setFormatter(formatter)
21
+ logger.addHandler(handler)
22
+ logger.propagate = False
23
+ return logger
@@ -0,0 +1,78 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ from concurrent.futures import ThreadPoolExecutor, as_completed
8
+ from pathlib import Path
9
+ from typing import List, Tuple
10
+
11
+ from tqdm import tqdm
12
+
13
+ from .config import MarkdownifyConfig
14
+ from .grouping import group_pages
15
+ from .llm import generate_markdown
16
+ from .logging import get_logger
17
+ from .pager import PageImage, load_document_pages
18
+ from .prompt_profiles import load_prompt_profile, PromptProfile
19
+
20
+ logger = get_logger("llm_markdownify.core")
21
+
22
+
23
+ class Markdownifier:
24
+ """Orchestrates the conversion of a document into Markdown using a Vision LLM."""
25
+
26
+ def __init__(self, config: MarkdownifyConfig, profile: str | None = None) -> None:
27
+ self.config = config
28
+ self.profile: PromptProfile = load_prompt_profile(profile or "contracts")
29
+
30
+ def _render_pages(self) -> List[PageImage]:
31
+ return load_document_pages(
32
+ self.config.input_path, dpi=self.config.dpi, allow_docx=self.config.allow_docx
33
+ )
34
+
35
+ def _group_pages(self, pages: List[PageImage]) -> List[List[PageImage]]:
36
+ return group_pages(
37
+ pages=pages,
38
+ model=self.config.model,
39
+ max_group_pages=self.config.max_group_pages,
40
+ enable_grouping=self.config.enable_grouping,
41
+ profile=self.profile,
42
+ )
43
+
44
+ def _markdown_for_group(self, group: List[PageImage]) -> str:
45
+ image_urls = [p.data_url for p in group]
46
+ return generate_markdown(
47
+ model=self.config.model,
48
+ image_data_urls=image_urls,
49
+ profile=self.profile,
50
+ temperature=self.config.temperature,
51
+ max_tokens=self.config.max_tokens,
52
+ )
53
+
54
+ def run(self) -> Path:
55
+ pages = self._render_pages()
56
+ groups = self._group_pages(pages)
57
+
58
+ logger.info(
59
+ "Processing %d groups with concurrency=%d", len(groups), self.config.concurrency
60
+ )
61
+
62
+ # Submit LLM work in parallel, but preserve order by collecting (idx, result)
63
+ results: List[Tuple[int, str]] = []
64
+ with ThreadPoolExecutor(max_workers=self.config.concurrency) as executor:
65
+ future_to_idx = {
66
+ executor.submit(self._markdown_for_group, group): idx
67
+ for idx, group in enumerate(groups)
68
+ }
69
+ for future in tqdm(as_completed(future_to_idx), total=len(groups), desc="LLM groups"):
70
+ idx = future_to_idx[future]
71
+ md = future.result()
72
+ results.append((idx, md))
73
+
74
+ ordered = [text for _, text in sorted(results, key=lambda t: t[0])]
75
+ output = "\n\n".join(ordered).strip() + "\n"
76
+ self.config.output_path.write_text(output, encoding="utf-8")
77
+ logger.info("Wrote Markdown to %s", self.config.output_path)
78
+ return self.config.output_path
@@ -0,0 +1,78 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ import base64
8
+ from dataclasses import dataclass
9
+ from io import BytesIO
10
+ from pathlib import Path
11
+ from typing import Iterable, List
12
+
13
+ import pypdfium2 as pdfium
14
+
15
+ from .logging import get_logger
16
+
17
+ try:
18
+ from docx2pdf import convert as docx2pdf_convert # type: ignore
19
+ except Exception: # pragma: no cover - optional
20
+ docx2pdf_convert = None # type: ignore
21
+
22
+ logger = get_logger("llm_markdownify.pager")
23
+
24
+
25
+ @dataclass
26
+ class PageImage:
27
+ index: int
28
+ width: int
29
+ height: int
30
+ content: bytes # PNG bytes
31
+
32
+ @property
33
+ def data_url(self) -> str:
34
+ b64 = base64.b64encode(self.content).decode("ascii")
35
+ return f"data:image/png;base64,{b64}"
36
+
37
+
38
+ def _docx_to_pdf(input_path: Path) -> Path:
39
+ if docx2pdf_convert is None:
40
+ raise RuntimeError(
41
+ "DOCX support requires 'docx2pdf' and platform support for Word/COM. Prefer PDFs."
42
+ )
43
+ temp_pdf = input_path.with_suffix(".converted.pdf")
44
+ logger.info("Converting DOCX to PDF: %s -> %s", input_path, temp_pdf)
45
+ docx2pdf_convert(str(input_path), str(temp_pdf))
46
+ return temp_pdf
47
+
48
+
49
+ def iter_pdf_pages_as_images(pdf_path: Path, dpi: int) -> Iterable[PageImage]:
50
+ logger.info("Rendering PDF pages to images at %s DPI", dpi)
51
+ pdf = pdfium.PdfDocument(str(pdf_path))
52
+ num_pages = len(pdf)
53
+ scale = dpi / 72.0
54
+ for i in range(num_pages):
55
+ page = pdf[i]
56
+ bitmap = page.render(scale=scale)
57
+ pil_image = bitmap.to_pil()
58
+ with BytesIO() as buf:
59
+ pil_image.save(buf, format="PNG")
60
+ data = buf.getvalue()
61
+ yield PageImage(index=i, width=pil_image.width, height=pil_image.height, content=data)
62
+
63
+
64
+ def load_document_pages(input_path: Path, dpi: int, allow_docx: bool = False) -> List[PageImage]:
65
+ """Load a PDF (preferred) or DOCX (if allowed) as a list of rendered page images."""
66
+ suffix = input_path.suffix.lower()
67
+ if suffix == ".docx":
68
+ if not allow_docx:
69
+ raise ValueError(
70
+ "DOCX not allowed. Prefer exporting DOCX to PDF, or enable --allow-docx (requires Word/COM)."
71
+ )
72
+ pdf_path = _docx_to_pdf(input_path)
73
+ else:
74
+ pdf_path = input_path
75
+
76
+ pages = list(iter_pdf_pages_as_images(pdf_path, dpi=dpi))
77
+ logger.info("Loaded %d pages", len(pages))
78
+ return pages
@@ -0,0 +1,81 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ import json
8
+ from dataclasses import dataclass
9
+ from pathlib import Path
10
+ from typing import Dict
11
+
12
+ from . import prompts as default_prompts
13
+
14
+
15
+ @dataclass(frozen=True)
16
+ class PromptProfile:
17
+ name: str
18
+ continuation_system: str
19
+ continuation_user: str
20
+ markdown_system: str
21
+ markdown_user: str
22
+
23
+
24
+ _BUILTIN_PROFILES: Dict[str, PromptProfile] = {
25
+ "contracts": PromptProfile(
26
+ name="contracts",
27
+ continuation_system=default_prompts.CONTINUATION_SYSTEM_PROMPT,
28
+ continuation_user=default_prompts.CONTINUATION_USER_PROMPT,
29
+ markdown_system=default_prompts.MARKDOWN_SYSTEM_PROMPT,
30
+ markdown_user=default_prompts.MARKDOWN_USER_PROMPT,
31
+ ),
32
+ "generic": PromptProfile(
33
+ name="generic",
34
+ continuation_system=(
35
+ "You analyze page images to decide if content visually continues (tables/charts split). "
36
+ "Respond only CONTINUE_NEXT or NONE."
37
+ ),
38
+ continuation_user=(
39
+ "Look for split tables/charts at the end of page A and start of page B. If found, CONTINUE_NEXT; else NONE."
40
+ ),
41
+ markdown_system=(
42
+ "Convert page images into clean Markdown. Use clear headings, lists, and tables; cover all pages; avoid page numbers."
43
+ ),
44
+ markdown_user=("Produce a single coherent Markdown output from the provided page images."),
45
+ ),
46
+ }
47
+
48
+
49
+ def load_prompt_profile(name_or_path: str) -> PromptProfile:
50
+ candidate = Path(name_or_path)
51
+ if candidate.exists() and candidate.is_file():
52
+ with candidate.open("r", encoding="utf-8") as f:
53
+ data = json.load(f)
54
+ required = {
55
+ "name",
56
+ "continuation_system",
57
+ "continuation_user",
58
+ "markdown_system",
59
+ "markdown_user",
60
+ }
61
+ missing = required - set(data.keys())
62
+ if missing:
63
+ raise ValueError(
64
+ f"Prompt profile missing required fields {sorted(missing)} in {candidate}"
65
+ )
66
+ return PromptProfile(
67
+ name=str(data["name"]),
68
+ continuation_system=str(data["continuation_system"]),
69
+ continuation_user=str(data["continuation_user"]),
70
+ markdown_system=str(data["markdown_system"]),
71
+ markdown_user=str(data["markdown_user"]),
72
+ )
73
+
74
+ key = name_or_path.strip().lower()
75
+ if key in _BUILTIN_PROFILES:
76
+ return _BUILTIN_PROFILES[key]
77
+
78
+ raise ValueError(
79
+ f"Unknown prompt profile '{name_or_path}'. Provide a built-in name ({', '.join(sorted(_BUILTIN_PROFILES))}) "
80
+ f"or a path to a JSON file with the fields: name, continuation_system, continuation_user, markdown_system, markdown_user."
81
+ )
@@ -0,0 +1,78 @@
1
+ # Copyright (c) 2025 Sethu Pavan Venkata Reddy Pastula
2
+ # Licensed under the Apache License, Version 2.0. See LICENSE file for details.
3
+ # SPDX-License-Identifier: Apache-2.0
4
+
5
+ from __future__ import annotations
6
+
7
+ CONTINUATION_SYSTEM_PROMPT = (
8
+ "You are a document structure analyst. Given one or two consecutive page images, "
9
+ "decide if the FIRST page should be MERGED with the NEXT page. \n"
10
+ "MERGE ONLY IF there is a clear visual continuation across the page boundary of: \n"
11
+ "- a table (gridlines/cell borders/columns cut at the bottom of page A and resuming at the top of page B), or\n"
12
+ "- a boxed layout/panel, or\n"
13
+ "- a chart/figure/diagram clearly split across pages.\n"
14
+ "Plain text paragraphs DO NOT qualify. Headings and body text continuity alone is NOT a reason to merge.\n"
15
+ "If uncertain, respond NONE. Respond with ONLY one token: CONTINUE_NEXT or NONE."
16
+ )
17
+
18
+ CONTINUATION_USER_PROMPT = (
19
+ "Check for split visual structures at the bottom of page A and the top of page B: \n"
20
+ "- table gridlines or cell borders continuing,\n"
21
+ "- boxed panels continuing,\n"
22
+ "- charts/figures cut between pages.\n"
23
+ "If you see such a split visual structure, answer CONTINUE_NEXT. Otherwise answer NONE."
24
+ )
25
+
26
+ MARKDOWN_SYSTEM_PROMPT = (
27
+ "You are a meticulous technical writer converting page images into clean Markdown. Follow this CHECKLIST strictly.\n"
28
+ "\n"
29
+ "1) Table of Contents (TOC) detection:\n"
30
+ " - If the page shows a TOC (label like 'CONTENTS' or dense sequential entries like '1 …', '2 …' with dot leaders/page numbers), render it as a simple list.\n"
31
+ " - DO NOT promote TOC entries to headings. Schedules in TOC are listed plainly.\n"
32
+ "\n"
33
+ "2) Top-level sections ('# '):\n"
34
+ " - A top-level section must be a concise, title-like line (typically ≤ 2 lines and a short phrase), not a long sentence.\n"
35
+ " - Use '# ' when the line begins with a single integer section number, optionally followed by a dot (e.g., '1', '1.', '12', '12.'), AND the remainder looks like a section name (short phrase, not a long sentence).\n"
36
+ " - Examples that SHOULD be headings: '# 2. SCOPE OF ENGAGEMENT', '# 18 ESCROW AGREEMENT'.\n"
37
+ " - Output as '# <full original line>' (preserve numeric prefix and casing).\n"
38
+ " - Even if an overall title exists (e.g., '# TERMS AND CONDITIONS'), subsequent integer sections still become '# 1 …', '# 2 …', etc.\n"
39
+ " - If the content after the number reads like a sentence (e.g., contains verbs such as 'shall', 'agree', 'will', 'must' early on, or clearly reads as a full clause), DO NOT make it a heading; keep it as a numbered line.\n"
40
+ "\n"
41
+ "3) Decimal-numbered items (NEVER headings):\n"
42
+ " - Items like '1.2', '12.1', '12.1.1' are NOT headings. Keep them as numbered lines under their parent section.\n"
43
+ " - If a page/group BEGINS with a decimal-numbered item (e.g., '4.3', '4.2.3'), treat it as CONTINUATION from a previous section; DO NOT fabricate '# 4'.\n"
44
+ " - Uppercase/bold/visual prominence does NOT change this rule. Decimal-numbered items must never become headings.\n"
45
+ "\n"
46
+ "4) Subheadings ('## '):\n"
47
+ " - Use '## ' ONLY for true subheadings that are NOT decimal-numbered list points (e.g., '## Warranty' under '# 3. Charges').\n"
48
+ " - Do not convert ordinary numbered paragraphs/lists into headings.\n"
49
+ "\n"
50
+ "5) Tables, charts, and images:\n"
51
+ " - Tables: render as valid GitHub-Flavored Markdown tables with proper alignment.\n"
52
+ " - Charts/diagrams: use Mermaid when feasible.\n"
53
+ " - Images/figures: include concise alt-text/captions. Dont try to provide a link to the image. You are just going to tell what the image is about in excruciating detail.\n"
54
+ "\n"
55
+ "6) Structure & hygiene:\n"
56
+ " - Cover ALL provided page images in order; do not omit later pages.\n"
57
+ " - Keep content readable; do not include page numbers or scanning artifacts.\n"
58
+ "\n"
59
+ "7) Output constraints (critical):\n"
60
+ " - Output ONLY the document content. DO NOT add any meta commentary, assurances, disclaimers, or notes (e.g., 'Note: Decimal-numbered items …', 'According to the images …', 'Summary of changes …').\n"
61
+ " - Do NOT include this checklist, instructions, or any explanation of your process.\n"
62
+ " - Do NOT add content that is not visibly present in the document.\n"
63
+ "\n"
64
+ "Examples (Do/Don't):\n"
65
+ "- Do: '# TERMS AND CONDITIONS' then '# 1. DEFINITIONS AND INTERPRETATION' then lines '1.1 …', '1.2 …'.\n"
66
+ "- Do: '# 2. SCOPE OF ENGAGEMENT' and '# 12 ESCROW AGREEMENT' (short, title-like).\n"
67
+ "- Do: At page start with '4.3 …', keep it as a numbered line (continuation), DO NOT add '# 4'.\n"
68
+ "- Do: TOC as a list of entries (no '#').\n"
69
+ "- Don't: '# 1.2 …' or '## 12.1 …' or '# 12.4 …'.\n"
70
+ "- Don't: '# 5 The Parties shall co-operate …' (this is a long, sentence-like numbered paragraph; keep as a numbered line).\n"
71
+ "- Don't: Any 'Note:'/'Disclaimer:' or statements about your output or method.\n"
72
+ )
73
+
74
+ MARKDOWN_USER_PROMPT = (
75
+ "Convert the provided page images into a single coherent Markdown segment. \n"
76
+ "Apply the checklist strictly: '# ' only for concise, title-like integer-numbered sections ('1'/'1.', '12'/'12.'); decimal-numbered items never become headings (even if bold/uppercase/first line); if the text after an integer number reads as a long sentence, keep it as a numbered line; TOC entries are lists; '## ' reserved only for true non-numbered subheadings. \n"
77
+ "Return ONLY the document content in Markdown with no notes, disclaimers, or extra commentary."
78
+ )