opendataloader-pdf 0.0.8__py3-none-any.whl → 0.0.9__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of opendataloader-pdf might be problematic. Click here for more details.

@@ -0,0 +1,395 @@
1
+ Metadata-Version: 2.4
2
+ Name: opendataloader-pdf
3
+ Version: 0.0.9
4
+ Summary: A Python wrapper for the opendataloader-pdf Java CLI.
5
+ Home-page: https://github.com/opendataloader-project/opendataloader-pdf
6
+ Author: opendataloader-project
7
+ Author-email: open.dataloader@hancom.com
8
+ License: MPL-2.0
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.7
12
+ Description-Content-Type: text/markdown
13
+ Dynamic: author
14
+ Dynamic: author-email
15
+ Dynamic: classifier
16
+ Dynamic: description
17
+ Dynamic: description-content-type
18
+ Dynamic: home-page
19
+ Dynamic: license
20
+ Dynamic: requires-python
21
+ Dynamic: summary
22
+
23
+ # OpenDataLoader PDF
24
+
25
+ ![Pre-release](https://img.shields.io/badge/Pre--release-FFA500&logo=github)
26
+ [![License](https://img.shields.io/pypi/l/opendataloader-pdf.svg)](https://pypi.org/project/opendataloader-pdf/)
27
+ [![Maven Central](https://img.shields.io/maven-central/v/io.github.opendataloader-project/opendataloader-pdf-core.svg)](https://search.maven.org/artifact/io.github.opendataloader-project/opendataloader-pdf-core)
28
+ [![PyPI version](https://img.shields.io/pypi/v/opendataloader-pdf.svg)](https://pypi.org/project/opendataloader-pdf/)
29
+ [![Python Version](https://img.shields.io/pypi/pyversions/opendataloader-pdf.svg)](https://pypi.org/project/opendataloader-pdf/)
30
+ [![GHCR Version](https://ghcr-badge.egpl.dev/opendataloader-project/opendataloader-pdf-cli/latest_tag?trim=major&label=docker-image)](https://github.com/opendataloader-project/opendataloader-pdf/pkgs/container/opendataloader-pdf-cli)
31
+ [![CLA assistant](https://cla-assistant.io/readme/badge/opendataloader-project/opendataloader-pdf)](https://cla-assistant.io/opendataloader-project/opendataloader-pdf)
32
+
33
+ <br/>
34
+
35
+ OpenDataLoader PDF converts PDFs into JSON, Markdown or Html — ready to feed into modern AI stacks (LLMs, vector search, and RAG).
36
+ It reconstructs document layout (headings, lists, tables, and reading order) so the content is easier to chunk, index, and query.
37
+ Powered by fast, heuristic, rule-based inference, it runs entirely on your local machine and delivers high-throughput processing for large document sets.
38
+ AI-safety is enabled by default and automatically filters likely prompt-injection content embedded in PDFs to reduce downstream risk.
39
+
40
+ <br/>
41
+
42
+ ## 🌟 Key Features
43
+
44
+ - 🧾 **Rich, structured output** — JSON, Markdown or Html
45
+ - 🧩 **Layout reconstruction** — headings, lists, tables, images, reading order
46
+ - 🔒 **Local-first privacy** — runs fully on your machine
47
+ - ⚡ **Fast & lightweight** — rule-based heuristic, high-throughput, no GPU
48
+ - 🛡️ **AI-safety by default** — auto-filters likely prompt-injection content
49
+ - 🆓 **Open source for business** — free for commercial use; Java/Python/Docker
50
+ - 🖍️ **Annotated PDF visualization** — see detected structures overlaid on the original
51
+
52
+ ![Annotated PDF Example](https://raw.githubusercontent.com/opendataloader-project/opendataloader-pdf/main/resources/example_annotated_pdf.png)
53
+
54
+ <br/>
55
+
56
+ ## 🚀 Upcoming Features
57
+
58
+ - 🖨️ **OCR for scanned PDFs** — image-only pages → selectable text
59
+ - 🧠 **Table AI option** — higher accuracy for borderless/merged cells
60
+ - 📊 **Layout benchmarks** — public datasets & metrics; regular reports
61
+ - 🛡️ **AI-Safety red-team** — adversarial datasets & metrics; regular reports
62
+
63
+ <br/>
64
+
65
+ ## Prerequisites
66
+
67
+ - Java 11 or higher must be installed and available in your system's PATH.
68
+ - Python 3.8+
69
+
70
+ <br/>
71
+
72
+ ## Python
73
+
74
+ ### Installation
75
+
76
+ ```sh
77
+ pip install opendataloader-pdf
78
+ ```
79
+
80
+ ### Usage
81
+
82
+ - input_path can be either the path to a single document or the path to a folder.
83
+ - If you don’t specify an output_folder, the output data will be saved in the same directory as the input document.
84
+
85
+ ```python
86
+ import opendataloader_pdf
87
+
88
+ opendataloader_pdf.run(
89
+ input_path="path/to/document.pdf",
90
+ output_folder="path/to/output",
91
+ to_markdown=True
92
+ )
93
+ ```
94
+
95
+ ### Function: run()
96
+
97
+ The main function to process PDFs.
98
+
99
+ | Parameter | Type | Required | Default | Description |
100
+ | ----------------------- | ------ | -------- | ------------ | --------------------------------------------------------------- |
101
+ | `input_path` | `str` | ✅ Yes | — | Path to the input PDF file or folder. |
102
+ | `output_folder` | `str` | No | input folder | Path to the output folder. |
103
+ | `password` | `str` | No | `None` | Password for the PDF file. |
104
+ | `to_markdown` | `bool` | No | `False` | If `True`, generates a Markdown output file. |
105
+ | `to_annotated_pdf` | `bool` | No | `False` | If `True`, generates an annotated PDF output file. |
106
+ | `keep_line_breaks` | `bool` | No | `False` | If `True`, keeps line breaks in the output. |
107
+ | `find_hidden_text` | `bool` | No | `False` | If `True`, finds hidden text in the PDF. |
108
+ | `html_in_markdown` | `bool` | No | `False` | If `True`, uses HTML in the Markdown output. |
109
+ | `add_image_to_markdown` | `bool` | No | `False` | If `True`, adds images to the Markdown output. |
110
+ | `debug` | `bool` | No | `False` | If `True`, prints CLI messages to the console during execution. |
111
+
112
+ <br/>
113
+
114
+ ## Java
115
+
116
+ ### Dependency
117
+
118
+ To include OpenDataLoader PDF in your Maven project, add the dependency below to your `pom.xml` file.
119
+
120
+ ```xml
121
+ <dependency>
122
+ <groupId>io.github.opendataloader-project</groupId>
123
+ <artifactId>opendataloader-pdf-core</artifactId>
124
+ <version>0.0.9</version>
125
+ </dependency>
126
+
127
+ <repositories>
128
+ <repository>
129
+ <snapshots>
130
+ <enabled>true</enabled>
131
+ </snapshots>
132
+ <id>vera-dev</id>
133
+ <name>Vera development</name>
134
+ <url>https://artifactory.openpreservation.org/artifactory/vera-dev</url>
135
+ </repository>
136
+ </repositories>
137
+ <pluginRepositories>
138
+ <pluginRepository>
139
+ <snapshots>
140
+ <enabled>false</enabled>
141
+ </snapshots>
142
+ <id>vera-dev</id>
143
+ <name>Vera development</name>
144
+ <url>https://artifactory.openpreservation.org/artifactory/vera-dev</url>
145
+ </pluginRepository>
146
+ </pluginRepositories>
147
+ ```
148
+
149
+
150
+ ### Java code integration
151
+
152
+ To integrate Layout recognition API into Java code, one can follow the sample code below.
153
+
154
+ ```java
155
+ import com.hancom.opendataloader.pdf.processors.DocumentProcessor;
156
+ import com.hancom.opendataloader.pdf.utils.Config;
157
+
158
+ import java.io.IOException;
159
+
160
+ public class Sample {
161
+
162
+ public static void main(String[] args) {
163
+ //create default config
164
+ Config config = new Config();
165
+
166
+ //set output folder relative to the input PDF
167
+ //if the output folder is not set, the current folder of the input PDF is used
168
+ config.setOutputFolder("output");
169
+
170
+ //generating pdf output file
171
+ config.setGeneratePDF(true);
172
+
173
+ //set password of input pdf file
174
+ config.setPassword("password");
175
+
176
+ //generate markdown output file
177
+ config.setGenerateMarkdown(true);
178
+
179
+ //enable html in markdown output file
180
+ config.setUseHTMLInMarkdown(true);
181
+
182
+ //add images to markdown output file
183
+ config.setAddImageToMarkdown(true);
184
+
185
+ //disable json output file
186
+ config.setGenerateJSON(false);
187
+
188
+ //keep line breaks
189
+ config.setKeepLineBreaks(true);
190
+
191
+ //find hidden text
192
+ config.setFindHiddenText(true);
193
+
194
+ try {
195
+ //process pdf file
196
+ DocumentProcessor.processFile("input.pdf",config);
197
+ } catch (Exception exception) {
198
+ //exception during processing
199
+ }
200
+ }
201
+ }
202
+ ```
203
+
204
+ <br/>
205
+
206
+ ## Docker
207
+
208
+ ```sh
209
+ # Download sample PDF
210
+ curl -L -o 1901.03003.pdf https://arxiv.org/pdf/1901.03003
211
+
212
+ # Run OpenDataLoader PDF in Docker container
213
+ docker run --rm -v "$PWD":/work ghcr.io/opendataloader-project/opendataloader-pdf-cli:latest /work/1901.03003.pdf --markdown --pdf
214
+ ```
215
+
216
+ <br/>
217
+
218
+ ## Developing with OpenDataLoader PDF
219
+
220
+ ### Build
221
+
222
+ Build and package using Maven command:
223
+
224
+ ```sh
225
+ mvn clean package -f java/pom.xml
226
+ ```
227
+
228
+ If the build is successful, the resulting `jar` file will be created in the path below.
229
+
230
+ ```sh
231
+ java/opendataloader-pdf-cli/target
232
+ ```
233
+
234
+ ### CLI usage
235
+
236
+ ```sh
237
+ java -jar ... [options] <INPUT FILE OR FOLDER>
238
+ ```
239
+
240
+ This generates a JSON file with layout recognition results in the specified output folder.
241
+ Additionally, annotated PDF with recognized structures and Markdown file are generated if options `--pdf` and `--markdown` are specified.
242
+
243
+ By default all line breaks and hyphenation characters are removed, the Markdown does not include any images and does not use any HTML.
244
+
245
+ The option `--keeplinebreaks` to preserve the original line breaks text content in JSON and Markdown output.
246
+
247
+ The option `--html`` enables use of HTML in Markdown, which may improve Markdown preview in processors that support HTML tags.
248
+ The option `--addimagetomarkdown` enables inclusion of image references into the output Markdown.
249
+ The images are extracted from PDF as individual files and stored in a subfolder next to the Markdown output.
250
+
251
+ #### Available options:
252
+
253
+ ```
254
+ Options:
255
+ -f,--folder <arg> Specify output folder (default the folder of the input PDF)
256
+ -klb,--keeplinebreaks Keep line breaks
257
+ -ht,--findhiddentext Find hidden text
258
+ -html,--htmlinmarkdown Use html in markdown
259
+ -im,--addimagetomarkdown Add images to markdown
260
+ -markdown,--markdown Generates markdown output
261
+ -p,--password <arg> Specifies password
262
+ -pdf,--pdf Generates pdf output
263
+ ```
264
+
265
+ ### Schema of the JSON output
266
+
267
+ Root json node
268
+
269
+ | Field | Type | Optional | Description |
270
+ |-------------------|---------|----------|------------------------------------|
271
+ | file name | string | no | Name of processed pdf file |
272
+ | number of pages | integer | no | Number of pages in pdf file |
273
+ | author | string | no | Author of pdf file |
274
+ | title | string | no | Title of pdf file |
275
+ | creation date | string | no | Creation date of pdf file |
276
+ | modification date | string | no | Modification date of pdf file |
277
+ | kids | array | no | Array of detected content elements |
278
+
279
+ Common fields of content json nodes
280
+
281
+ | Field | Type | Optional | Description |
282
+ |--------------|---------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
283
+ | id | integer | yes | Unique id of content element |
284
+ | level | string | yes | Level of content element |
285
+ | type | string | no | Type of content element<br/>Possible types: `footer`, `header`, `heading`, `line`, `table`, `table row`, `table cell`, `paragraph`, `list`, `list item`, `image`, `line art`, `caption`, `text block` |
286
+ | page number | integer | no | Page number of content element |
287
+ | bounding box | array | no | Bounding box of content element |
288
+
289
+ Specific fields of text content json nodes (`caption`, `heading`, `paragraph`)
290
+
291
+ | Field | Type | Optional | Description |
292
+ |------------|--------|----------|-------------------|
293
+ | font | string | no | Font name of text |
294
+ | font size | double | no | Font size of text |
295
+ | text color | array | no | Color of text |
296
+ | content | string | no | Text value |
297
+
298
+ Specific fields of `table` json nodes
299
+
300
+ | Field | Type | Optional | Description |
301
+ |-------------------|---------|----------|--------------------------------|
302
+ | number of rows | integer | no | Number of table rows |
303
+ | number of columns | integer | no | Number of table columns |
304
+ | rows | array | no | Array of table rows |
305
+ | previous table id | integer | yes | Id of previous connected table |
306
+ | next table id | integer | yes | Id of next connected table |
307
+
308
+ Specific fields of `table row` json nodes
309
+
310
+ | Field | Type | Optional | Description |
311
+ |------------|---------|----------|----------------------|
312
+ | row number | integer | no | Number of table row |
313
+ | cells | array | no | Array of table cells |
314
+
315
+ Specific fields of `table cell` json nodes
316
+
317
+ | Field | Type | Optional | Description |
318
+ |---------------|---------|----------|--------------------------------------|
319
+ | row number | integer | no | Row number of table cell |
320
+ | column number | integer | no | Column number of table cell |
321
+ | row span | integer | no | Row span of table cell |
322
+ | column span | integer | no | Column span of table cell |
323
+ | kids | array | no | Array of table cell content elements |
324
+
325
+ Specific fields of `heading` json nodes
326
+
327
+ | Field | Type | Optional | Description |
328
+ |---------------|---------|----------|--------------------------|
329
+ | heading level | integer | no | Heading level of heading |
330
+
331
+ Specific fields of `list` json nodes
332
+
333
+ | Field | Type | Optional | Description |
334
+ |----------------------|---------|----------|-------------------------------------|
335
+ | number of list items | integer | no | Number of list items |
336
+ | numbering style | string | no | Numbering style of this list |
337
+ | previous list id | integer | yes | Id of previous connected list |
338
+ | next list id | integer | yes | Id of next connected list |
339
+ | list items | array | no | Array of list item content elements |
340
+
341
+ Specific fields of `list item` json nodes
342
+
343
+ | Field | Type | Optional | Description |
344
+ |-------|-------|----------|-------------------------------------|
345
+ | kids | array | no | Array of list item content elements |
346
+
347
+ Specific fields of `header` and `footer` json nodes
348
+
349
+ | Field | Type | Optional | Description |
350
+ |-------|-------|----------|-----------------------------------------|
351
+ | kids | array | no | Array of header/footer content elements |
352
+
353
+ Specific fields of `text block` json nodes
354
+
355
+ | Field | Type | Optional | Description |
356
+ |-------|-------|----------|--------------------------------------|
357
+ | kids | array | no | Array of text block content elements |
358
+
359
+
360
+ ## 🤝 Contributing
361
+
362
+ We believe that great software is built together.
363
+
364
+ Your contributions are vital to the success of this project.
365
+
366
+ Please read [CONTRIBUTING.md](https://github.com/hancom-inc/opendataloader-pdf/blob/main/CONTRIBUTING.md) for details on how to contribute.
367
+
368
+ ## 💖 Community & Support
369
+ Have questions or need a little help? We're here for you!🤗
370
+
371
+ - [GitHub Discussions](https://github.com/hancom-inc/opendataloader-pdf/discussions): For Q&A and general chats. Let's talk! 🗣️
372
+ - [GitHub Issues](https://github.com/hancom-inc/opendataloader-pdf/issues): Found a bug? 🐛 Please report it here so we can fix it.
373
+
374
+ ## ✨ Our Branding and Trademarks
375
+
376
+ We love our brand and want to protect it!
377
+
378
+ This project may contain trademarks, logos, or brand names for our products and services.
379
+
380
+ To ensure everyone is on the same page, please remember these simple rules:
381
+
382
+ - **Authorized Use**: You're welcome to use our logos and trademarks, but you must follow our official brand guidelines.
383
+ - **No Confusion**: When you use our trademarks in a modified version of this project, it should never cause confusion or imply that Hancom officially sponsors or endorses your version.
384
+ - **Third-Party Brands**: Any use of trademarks or logos from other companies must follow that company’s specific policies.
385
+
386
+ ## ⚖️ License
387
+
388
+ This project is licensed under the [Mozilla Public License 2.0](https://www.mozilla.org/MPL/2.0/).
389
+
390
+ For the full license text, see [LICENSE](LICENSE).
391
+
392
+ For information on third-party libraries and components, see:
393
+ - [THIRD_PARTY_LICENSES](./THIRD_PARTY/THIRD_PARTY_LICENSES.md)
394
+ - [THIRD_PARTY_NOTICES](./THIRD_PARTY/THIRD_PARTY_NOTICES.md)
395
+ - [licenses/](./THIRD_PARTY/licenses/)
@@ -13,8 +13,8 @@ opendataloader_pdf/THIRD_PARTY/licenses/LICENSE-JJ2000.txt,sha256=itSesIy3XiNWgJ
13
13
  opendataloader_pdf/THIRD_PARTY/licenses/MIT.txt,sha256=JPCdbR3BU0uO_KypOd3sGWnKwlVHGq4l0pmrjoGtop8,1078
14
14
  opendataloader_pdf/THIRD_PARTY/licenses/MPL-2.0.txt,sha256=CGF6Fx5WV7DJmRZJ8_6w6JEt2N9bu4p6zDo18fTHHRw,15818
15
15
  opendataloader_pdf/THIRD_PARTY/licenses/Plexus Classworlds License.txt,sha256=ZQuKXwVz4FeC34ApB20vYg8kPTwgIUKRzEk5ew74-hU,1937
16
- opendataloader_pdf/jar/opendataloader-pdf-cli.jar,sha256=PHFGUXwEEbHmPTjS72-zvDZIOYbHSveceBjkT_ovYtA,22114523
17
- opendataloader_pdf-0.0.8.dist-info/METADATA,sha256=JGSMFKtXQsMW1T2-9lgwHdfPJjk11BMpYMH6I_SoEeg,2352
18
- opendataloader_pdf-0.0.8.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
19
- opendataloader_pdf-0.0.8.dist-info/top_level.txt,sha256=xee0qFQd6HPfS50E2NLICGuR6cq9C9At5SJ81yv5HkY,19
20
- opendataloader_pdf-0.0.8.dist-info/RECORD,,
16
+ opendataloader_pdf/jar/opendataloader-pdf-cli.jar,sha256=ZtctQzzoQpnKCxEM4H6ktVTw4l5ZbP5gAetbPgp_bmA,22114701
17
+ opendataloader_pdf-0.0.9.dist-info/METADATA,sha256=SVLovntCpxWdnlpsXwg0E1i3ct_uq-WchM4ximmY_m0,17559
18
+ opendataloader_pdf-0.0.9.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
19
+ opendataloader_pdf-0.0.9.dist-info/top_level.txt,sha256=xee0qFQd6HPfS50E2NLICGuR6cq9C9At5SJ81yv5HkY,19
20
+ opendataloader_pdf-0.0.9.dist-info/RECORD,,
@@ -1,67 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: opendataloader-pdf
3
- Version: 0.0.8
4
- Summary: A Python wrapper for the opendataloader-pdf Java CLI.
5
- Home-page: https://github.com/opendataloader-project/opendataloader-pdf
6
- Author: opendataloader-project
7
- Author-email: open.dataloader@hancom.com
8
- License: MPL-2.0
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.7
12
- Description-Content-Type: text/markdown
13
- Dynamic: author
14
- Dynamic: author-email
15
- Dynamic: classifier
16
- Dynamic: description
17
- Dynamic: description-content-type
18
- Dynamic: home-page
19
- Dynamic: license
20
- Dynamic: requires-python
21
- Dynamic: summary
22
-
23
- # Opendataloader PDF Python Wrapper
24
-
25
- This package is a Python wrapper for the `opendataloader-pdf` Java command-line tool.
26
-
27
- It allows you to process PDF files and convert them to JSON or Markdown format directly from Python.
28
-
29
- ## Prerequisites
30
-
31
- - Java 11 or higher must be installed and available in your system's PATH.
32
-
33
- ## Installation
34
-
35
- ```bash
36
- pip install opendataloader-pdf
37
- ```
38
-
39
- ## Usage
40
-
41
- Here is a basic example of how to use:
42
-
43
- ```python
44
- import opendataloader_pdf
45
-
46
- opendataloader_pdf.run("path/to/document.pdf", to_markdown=True)
47
-
48
- # If you don’t specify an output_folder,
49
- # the output data will be saved in the same directory as the input document.
50
- ```
51
-
52
- ## Function: `run()`
53
-
54
- The main function to process PDFs.
55
-
56
- ### Parameters
57
-
58
- - `input_path` (str): Path to the input PDF file or folder. **(Required)**
59
- - `output_folder` (str, optional): Path to the output folder. Defaults to the input folder.
60
- - `password` (str, optional): Password for the PDF file.
61
- - `to_markdown` (bool, optional): If `True`, generates a Markdown output file. Defaults to `False`.
62
- - `to_annotated_pdf` (bool, optional): If `True`, generates an annotated PDF output file. Defaults to `False`.
63
- - `keep_line_breaks` (bool, optional): If `True`, keeps line breaks in the output. Defaults to `False`.
64
- - `find_hidden_text` (bool, optional): If `True`, finds hidden text in the PDF. Defaults to `False`.
65
- - `html_in_markdown` (bool, optional): If `True`, uses HTML in the Markdown output. Defaults to `False`.
66
- - `add_image_to_markdown` (bool, optional): If `True`, adds images to the Markdown output. Defaults to `False`.
67
- - `debug` (bool, optional): If `True`, prints all messages from the CLI to the console during execution. Defaults to `False`.