grobid-client-python 0.0.17__tar.gz → 0.1.0__tar.gz

{grobid_client_python-0.0.17 → grobid_client_python-0.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: grobid-client-python
-Version: 0.0.17
+Version: 0.1.0
 Summary: Simple python client for GROBID REST services
 Author-email: Patrice Lopez <patrice.lopez@science-miner.com>
 Maintainer-email: Patrice Lopez <patrice.lopez@science-miner.com>, Luca Foppiano <lucanoro@duck.com>
@@ -213,6 +213,9 @@ Project-URL: Repository, https://github.com/kermitt2/grobid_client_python
 Project-URL: Changelog, https://github.com/kermitt2/grobid_client_python
 Description-Content-Type: text/markdown
 Requires-Dist: requests
+Requires-Dist: dateparser
+Requires-Dist: beautifulsoup4
+Requires-Dist: lxml
 
 # GROBID Client Python
 
@@ -220,7 +223,8 @@ Requires-Dist: requests
 [![SWH](https://archive.softwareheritage.org/badge/origin/https://github.com/kermitt2/grobid_client_python/)](https://archive.softwareheritage.org/browse/origin/https://github.com/kermitt2/grobid_client_python/)
 [![License](http://img.shields.io/:license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)
 
-A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobid) REST services that provides concurrent processing capabilities for PDF documents, reference strings, and patents.
+A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobid) REST services that provides
+concurrent processing capabilities for PDF documents, reference strings, and patents.
 
 ## 📋 Table of Contents
 
@@ -229,8 +233,8 @@ A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobi
 - [Installation](#-installation)
 - [Quick Start](#-quick-start)
 - [Usage](#-usage)
-  - [Command Line Interface](#command-line-interface)
-  - [Python Library](#python-library)
+    - [Command Line Interface](#command-line-interface)
+    - [Python Library](#python-library)
 - [Configuration](#-configuration)
 - [Services](#-services)
 - [Testing](#-testing)
@@ -246,34 +250,42 @@ A simple, efficient Python client for [GROBID](https://github.com/kermitt2/grobi
 - **Command Line & Library**: Use as a standalone CLI tool or import into your Python projects
 - **Coordinate Extraction**: Optional PDF coordinate extraction for precise element positioning
 - **Sentence Segmentation**: Layout-aware sentence segmentation capabilities
+- **JSON Output**: Convert TEI XML output to structured JSON format with CORD-19-like structure
+- **Markdown Output**: Convert TEI XML output to clean Markdown format with structured sections
 
 ## 📋 Prerequisites
 
 - **Python**: 3.8 - 3.13 (tested versions)
 - **GROBID Server**: A running GROBID service instance
-  - Local installation: [GROBID Documentation](http://grobid.readthedocs.io/)
-  - Docker: `docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.2`
-  - Default server: `http://localhost:8070`
-  - Online demo: https://lfoppiano-grobid.hf.space (usage limits apply), more details [here](https://grobid.readthedocs.io/en/latest/getting_started/#using-grobid-from-the-cloud).
+    - Local installation: [GROBID Documentation](http://grobid.readthedocs.io/)
+    - Docker: `docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.2`
+    - Default server: `http://localhost:8070`
+    - Online demo: https://lfoppiano-grobid.hf.space (usage limits apply), more
+      details [here](https://grobid.readthedocs.io/en/latest/getting_started/#using-grobid-from-the-cloud).
+
 
 > [!IMPORTANT]
-> GROBID supports Windows only through Docker containers. See the [Docker documentation](https://grobid.readthedocs.io/en/latest/Grobid-docker/) for details.
+> GROBID supports Windows only through Docker containers. See
+> the [Docker documentation](https://grobid.readthedocs.io/en/latest/Grobid-docker/) for details.
 
 ## 🚀 Installation
 
 Choose one of the following installation methods:
 
 ### PyPI (Recommended)
+
 ```bash
 pip install grobid-client-python
 ```
 
 ### Development Version
+
 ```bash
 pip install git+https://github.com/kermitt2/grobid_client_python.git
 ```
 
 ### Local Development
+
 ```bash
 git clone https://github.com/kermitt2/grobid_client_python
 cd grobid_client_python
@@ -283,6 +295,7 @@ pip install -e .
 ## ⚡ Quick Start
 
 ### Command Line
+
 ```bash
 # Process PDFs in a directory
 grobid_client --input ./pdfs --output ./output processFulltextDocument
@@ -292,6 +305,7 @@ grobid_client --server https://your-grobid-server.com --input ./pdfs processFull
 ```
 
 ### Python Library
+
 ```python
 from grobid_client.grobid_client import GrobidClient
 
@@ -347,6 +361,9 @@ grobid_client [OPTIONS] SERVICE
 | `--teiCoordinates`           | Add PDF coordinates to XML                |
 | `--segmentSentences`         | Segment sentences with coordinates        |
 | `--flavor`                   | Processing flavor for fulltext extraction |
+| `--json`                     | Convert TEI output to JSON format         |
+| `--markdown`                 | Convert TEI output to Markdown format     |
+
 
 #### Examples
 
@@ -357,11 +374,17 @@ grobid_client --input ~/documents --output ~/results processFulltextDocument
 # High concurrency with coordinates
 grobid_client --input ~/pdfs --output ~/tei --n 20 --teiCoordinates processFulltextDocument
 
+# Process with JSON output
+grobid_client --input ~/pdfs --output ~/results --json processFulltextDocument
+
+# Process with Markdown output
+grobid_client --input ~/pdfs --output ~/results --markdown processFulltextDocument
+
 # Process citations with custom server
 grobid_client --server https://grobid.example.com --input ~/citations.txt processCitationList
 
-# Force reprocessing with sentence segmentation
-grobid_client --input ~/docs --force --segmentSentences processFulltextDocument
+# Force reprocessing with sentence segmentation and JSON output
+grobid_client --input ~/docs --force --segmentSentences --json processFulltextDocument
 ```
 
 ### Python Library
@@ -404,6 +427,22 @@ client.process(
     segmentSentences=True
 )
 
+# Process with JSON output
+client.process(
+    service="processFulltextDocument",
+    input_path="/path/to/pdfs",
+    output_path="/path/to/output",
+    json_output=True
+)
+
+# Process with Markdown output
+client.process(
+    service="processFulltextDocument",
+    input_path="/path/to/pdfs",
+    output_path="/path/to/output",
+    markdown_output=True
+)
+
 # Process citation lists
 client.process(
     service="processCitationList",
@@ -414,17 +453,25 @@ client.process(
 
 ## ⚙️ Configuration
 
-Configuration can be provided via a JSON file. When using the CLI, the `--server` argument overrides the config file settings.
+Configuration can be provided via a JSON file. When using the CLI, the `--server` argument overrides the config file
+settings.
 
 ### Default Configuration
 
 ```json
 {
-    "grobid_server": "http://localhost:8070",
-    "batch_size": 1000,
-    "sleep_time": 5,
-    "timeout": 60,
-    "coordinates": ["persName", "figure", "ref", "biblStruct", "formula", "s"]
+  "grobid_server": "http://localhost:8070",
+  "batch_size": 1000,
+  "sleep_time": 5,
+  "timeout": 60,
+  "coordinates": [
+    "persName",
+    "figure",
+    "ref",
+    "biblStruct",
+    "formula",
+    "s"
+  ]
 }
 ```
 
@@ -437,20 +484,267 @@ Configuration can be provided via a JSON file. When using the CLI, the `--server
 | `sleep_time`    | Wait time when server is busy (seconds)                                                                          | 5                       |
 | `timeout`       | Client-side timeout (seconds)                                                                                    | 180                     |
 | `coordinates`   | XML elements for coordinate extraction                                                                           | See above               |
+| `logging`       | Logging configuration (level, format, file output)                                                              | See Logging section     |
 
 > [!TIP]
-> Since version 0.0.12, the config file is optional. The client will use default localhost settings if no configuration is provided.
+> Since version 0.0.12, the config file is optional. The client will use default localhost settings if no configuration
+> is provided.
+
+### Logging Configuration
+
+The client provides configurable logging with different verbosity levels. By default, only essential statistics and warnings are shown.
+
+#### Logging Behavior
+
+- **Without `--verbose`**: Shows only essential information and warnings/errors
+- **With `--verbose`**: Shows detailed processing information at INFO level
+
+#### Always Visible Output
+
+The following information is always displayed regardless of the `--verbose` flag:
+
+```bash
+Found 1000 file(s) to process
+Processing completed: 950 out of 1000 files processed
+Errors: 50 out of 1000 files processed
+Processing completed in 120.5 seconds
+```
+
+#### Verbose Output (`--verbose`)
+
+When the `--verbose` flag is used, additional detailed information is displayed:
+
+- Server connection status
+- Individual file processing details
+- JSON conversion messages
+- Detailed error messages
+- Processing progress information
+
+#### Examples
+
+```bash
+# Clean output - only essential statistics
+grobid_client --input pdfs/ processFulltextDocument
+# Output:
+# Found 1000 file(s) to process
+# Processing completed: 950 out of 1000 files processed
+# Errors: 50 out of 1000 files processed
+# Processing completed in 120.5 seconds
+
+# Verbose output - detailed processing information
+grobid_client --input pdfs/ --verbose processFulltextDocument
+# Output includes all essential stats PLUS:
+# GROBID server http://localhost:8070 is up and running
+# JSON file example.json does not exist, generating JSON from existing TEI...
+# Successfully created JSON file: example.json
+# ... and other detailed processing information
+```
+
+#### Configuration File Logging
+
+The config file can include logging settings:
+
+```json
+{
+    "grobid_server": "http://localhost:8070",
+    "logging": {
+        "level": "WARNING",
+        "format": "%(asctime)s - %(levelname)s - %(message)s",
+        "console": true,
+        "file": null
+    }
+}
+```
+
+**Note**: The `--verbose` command line flag always takes precedence over configuration file logging settings.
 
 ## 🔬 Services
 
 ### Fulltext Document Processing
+
 Extracts complete document structure including headers, body text, figures, tables, and references.
 
 ```bash
 grobid_client --input pdfs/ --output results/ processFulltextDocument
 ```
 
+### JSON Output Format
+
+When using the `--json` flag, the client converts TEI XML output to a structured JSON format similar to CORD-19. This provides:
+
+- **Structured Bibliography**: Title, authors, DOI, publication date, journal information
+- **Body Text**: Paragraphs and sentences with metadata and reference annotations
+- **Figures and Tables**: Structured JSON format for tables with headers, rows, and metadata
+- **Reference Information**: In-text citations with offsets and targets
+
+#### JSON Structure
+
+```json
+{
+  "level": "paragraph",
+  "biblio": {
+    "title": "Document Title",
+    "authors": [
+      "Author 1",
+      "Author 2"
+    ],
+    "doi": "10.1000/example",
+    "publication_date": "2023-01-01",
+    "journal": "Journal Name",
+    "abstract": [
+      ...
+    ]
+  },
+  "body_text": [
+    {
+      "id": "p_12345",
+      "text": "Paragraph text with citations [1].",
+      "head_section": "Introduction",
+      "refs": [
+        {
+          "type": "bibr",
+          "target": "b1",
+          "text": "[1]",
+          "offset_start": 25,
+          "offset_end": 28
+        }
+      ]
+    }
+  ],
+  "figures_and_tables": [
+    {
+      "id": "table_1",
+      "type": "table",
+      "label": "Table 1",
+      "head": "Sample Data",
+      "content": {
+        "headers": [
+          "Header 1",
+          "Header 2"
+        ],
+        "rows": [
+          [
+            "Value 1",
+            "Value 2"
+          ]
+        ],
+        "metadata": {
+          "row_count": 1,
+          "column_count": 2,
+          "has_headers": true
+        }
+      }
+    }
+  ]
+}
+```
+
+#### Usage Examples
+
+```bash
+# Generate both TEI and JSON outputs
+grobid_client --input pdfs/ --output results/ --json processFulltextDocument
+
+# JSON output with coordinates and sentence segmentation
+grobid_client --input pdfs/ --output results/ --json --teiCoordinates --segmentSentences processFulltextDocument
+```
+
+```python
+# Python library usage
+client.process(
+    service="processFulltextDocument",
+    input_path="/path/to/pdfs",
+    output_path="/path/to/output",
+    json_output=True
+)
+```
+
+> [!NOTE]
+> When using `--json`, the `--force` flag only checks for existing TEI files. If a TEI file is rewritten (due to
+`--force`), the corresponding JSON file is automatically rewritten as well.
+
+### Markdown Output Format
+
+When using the `--markdown` flag, the client converts TEI XML output to a clean, readable Markdown format. This
+provides:
+
+- **Structured Sections**: Title, Authors, Affiliations, Publication Date, Fulltext, Annex, and References
+- **Clean Formatting**: Human-readable format suitable for documentation and sharing
+- **Preserved Content**: All text content with proper section organization
+- **Reference Formatting**: Bibliographic references in a readable format
+
+#### Markdown Structure
+
+The generated Markdown follows this structure:
+
+```markdown
+# Document Title
+
+## Authors
+
+- Author Name 1
+- Author Name 2
+
+## Affiliations
+
+- Affiliation 1
+- Affiliation 2
+
+## Publication Date
+
+January 1, 2023
+
+## Fulltext
+
+### Introduction
+
+Content of the introduction section...
+
+### Methods
+
+Content of the methods section...
+
+## Annex
+
+### Acknowledgements
+
+Acknowledgement text...
+
+### Competing Interests
+
+Competing interests statement...
+
+## References
+
+**[1]** Paper Title. *Author Name*. *Journal Name* (2023).
+**[2]** Another Paper. *Author et al.*. *Conference* (2022).
+```
+
+#### Usage Examples
+
+```bash
+# Generate both TEI and Markdown outputs
+grobid_client --input pdfs/ --output results/ --markdown processFulltextDocument
+
+# Markdown output with coordinates and sentence segmentation
+grobid_client --input pdfs/ --output results/ --markdown --teiCoordinates --segmentSentences processFulltextDocument
+```
+
+```python
+# Python library usage
+client.process(
+    service="processFulltextDocument",
+    input_path="/path/to/pdfs",
+    output_path="/path/to/output",
+    markdown_output=True
+)
+```
+
+> [!NOTE]
+> When using `--markdown`, the `--force` flag only checks for existing TEI files. If a TEI file is rewritten (due to `--force`), the corresponding Markdown file is automatically rewritten as well.
+
 ### Header Document Processing
+
 Extracts only document metadata (title, authors, abstract, etc.).
 
 ```bash
@@ -458,6 +752,7 @@ grobid_client --input pdfs/ --output headers/ processHeaderDocument
 ```
 
 ### Reference Processing
+
 Extracts and structures bibliographic references from documents.
 
 ```bash
@@ -465,6 +760,7 @@ grobid_client --input pdfs/ --output refs/ processReferences
 ```
 
 ### Citation List Processing
+
 Parses raw citation strings from text files.
 
 ```bash
@@ -507,6 +803,7 @@ pytest -v
 ### Continuous Integration
 
 Tests are automatically run via GitHub Actions on:
+
 - Push to main branch
 - Pull requests
 - Multiple Python versions (3.8-3.13)
@@ -529,7 +826,7 @@ Benchmark results for processing **136 PDFs** (3,443 pages total, ~25 pages per
 ### Additional Benchmarks
 
 - **Header processing**: 3.74s for 136 PDFs (36 PDF/s) with n=10
-- **Reference extraction**: 26.9s for 136 PDFs (5.1 PDF/s) with n=10  
+- **Reference extraction**: 26.9s for 136 PDFs (5.1 PDF/s) with n=10
 - **Citation parsing**: 4.3s for 3,500 citations (814 citations/s) with n=10
 
 ## 🛠️ Development
@@ -579,7 +876,8 @@ bump-my-version bump patch
 
 ## 📄 License
 
-Distributed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0). See `LICENSE` for more information.
+Distributed under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0). See `LICENSE` for more
+information.
 
 ## 👥 Authors & Contact