pdf-to-markdown-cli 0.4.0__tar.gz → 0.5.1__tar.gz

pdf_to_markdown_cli-0.5.1/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: pdf-to-markdown-cli
+Version: 0.5.1
+Summary: CLI tool to convert PDF files (and other documents) to markdown using the Marker API.
+Author: Nikita Sokolsky
+License: MIT License
+        
+        Copyright (c) 2025 SokolskyNikita
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/SokolskyNikita/pdf-to-markdown-cli
+Project-URL: Bug Tracker, https://github.com/SokolskyNikita/pdf-to-markdown-cli/issues
+Keywords: pdf,markdown,converter,cli,document,marker,md
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: backoff>=2.0
+Requires-Dist: diskcache>=5.0
+Requires-Dist: filetype>=1.0
+Requires-Dist: pikepdf>=8.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: ratelimit>=2.0
+Requires-Dist: requests>=2.0
+Requires-Dist: tqdm>=4.0
+Provides-Extra: test
+Dynamic: license-file
+
+# PDF to Markdown CLI
+
+Convert PDFs and other documents to Markdown using the [Marker API](https://www.datalab.to/marker).
+
+## Features
+
+- Convert PDFs, Word docs, PowerPoint, spreadsheets, EPUB, HTML, and images to Markdown/JSON/HTML
+- Automatic chunking for large documents with parallel processing
+- Progress tracking and local caching for interrupted runs
+- Full OCR customization options
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install pdf-to-markdown-cli
+```
+
+### From source
+
+```bash
+git clone https://github.com/SokolskyNikita/pdf-to-markdown-cli.git 
+cd pdf-to-markdown-cli
+pip install -e .
+```
+
+## Usage
+
+```bash
+# Get API key from https://www.datalab.to/marker
+export MARKER_PDF_KEY=your_api_key_here
+
+# Basic usage
+pdf-to-md /path/to/file.pdf
+
+# Common options
+pdf-to-md /path/to/file.pdf --json          # JSON output
+pdf-to-md /path/to/file.pdf --noimg         # Disable images  
+pdf-to-md /path/to/file.pdf --max           # Enable all flags for maximum output quality
+```
+
+## CLI Options
+
+- `input`: Input file or directory path
+- `--json`: Output in JSON format (default is markdown)
+- `--langs`: Comma-separated OCR languages (default: "English")
+- `--llm`: Use LLM for enhanced processing
+- `--strip`: Redo OCR processing
+- `--noimg`: Disable image extraction
+- `--force`: Force OCR on all pages
+- `--pages`: Add page delimiters
+- `--max`: Enable all OCR enhancements (equivalent to --llm --strip --force)
+- `-mp`, `--max-pages`: Maximum number of pages to process from the start of the file
+- `--no-chunk`: Disable PDF chunking
+- `-cs`, `--chunk-size`: Set PDF chunk size in pages (default: 25)
+- `-o`, `--output-dir`: Absolute path to the output directory
+- `-v`, `--verbose`: Enable verbose (DEBUG level) logging
+- `--version`: Show the installed version and exit
+
+## Requirements
+
+- Python ≥3.10
+- API key from [datalab.to](https://www.datalab.to/marker)

pdf_to_markdown_cli-0.5.1/README.md

@@ -0,0 +1,64 @@
+# PDF to Markdown CLI
+
+Convert PDFs and other documents to Markdown using the [Marker API](https://www.datalab.to/marker).
+
+## Features
+
+- Convert PDFs, Word docs, PowerPoint, spreadsheets, EPUB, HTML, and images to Markdown/JSON/HTML
+- Automatic chunking for large documents with parallel processing
+- Progress tracking and local caching for interrupted runs
+- Full OCR customization options
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install pdf-to-markdown-cli
+```
+
+### From source
+
+```bash
+git clone https://github.com/SokolskyNikita/pdf-to-markdown-cli.git 
+cd pdf-to-markdown-cli
+pip install -e .
+```
+
+## Usage
+
+```bash
+# Get API key from https://www.datalab.to/marker
+export MARKER_PDF_KEY=your_api_key_here
+
+# Basic usage
+pdf-to-md /path/to/file.pdf
+
+# Common options
+pdf-to-md /path/to/file.pdf --json          # JSON output
+pdf-to-md /path/to/file.pdf --noimg         # Disable images  
+pdf-to-md /path/to/file.pdf --max           # Enable all flags for maximum output quality
+```
+
+## CLI Options
+
+- `input`: Input file or directory path
+- `--json`: Output in JSON format (default is markdown)
+- `--langs`: Comma-separated OCR languages (default: "English")
+- `--llm`: Use LLM for enhanced processing
+- `--strip`: Redo OCR processing
+- `--noimg`: Disable image extraction
+- `--force`: Force OCR on all pages
+- `--pages`: Add page delimiters
+- `--max`: Enable all OCR enhancements (equivalent to --llm --strip --force)
+- `-mp`, `--max-pages`: Maximum number of pages to process from the start of the file
+- `--no-chunk`: Disable PDF chunking
+- `-cs`, `--chunk-size`: Set PDF chunk size in pages (default: 25)
+- `-o`, `--output-dir`: Absolute path to the output directory
+- `-v`, `--verbose`: Enable verbose (DEBUG level) logging
+- `--version`: Show the installed version and exit
+
+## Requirements
+
+- Python ≥3.10
+- API key from [datalab.to](https://www.datalab.to/marker)

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pdf-to-markdown-cli"
-version = "0.4.0"
+version = "0.5.1"
 description = "CLI tool to convert PDF files (and other documents) to markdown using the Marker API."
 readme = "README.md"
 authors = [
@@ -48,4 +48,7 @@ pdf-to-md = "docs_to_md.main:main"
 
 # Just in case: Configure setuptools to find packages in src/
 [tool.setuptools.packages.find]
-where = ["src"]
+where = ["src"]
+
+[project.optional-dependencies]
+test = []

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/docs_to_md/api/client.py

@@ -73,14 +73,24 @@ class MarkerClient:
             form_data = {
                 "file": (file_path.name, file_data, kind.mime),
                 "langs": (None, langs),
-                "force_ocr": (None, force_ocr),
-                "paginate": (None, paginate),
-                "strip_existing_ocr": (None, strip_existing_ocr),
-                "disable_image_extraction": (None, disable_image_extraction),
-                "use_llm": (None, use_llm),
                 "output_format": (None, output_format),
-                "max_pages": (None, max_pages),
             }
+            
+            # Add boolean parameters with proper serialization
+            if force_ocr:
+                form_data["force_ocr"] = (None, "true")
+            if paginate:
+                form_data["paginate"] = (None, "true")
+            if strip_existing_ocr:
+                form_data["strip_existing_ocr"] = (None, "true")
+            if disable_image_extraction:
+                form_data["disable_image_extraction"] = (None, "true")
+            if use_llm:
+                form_data["use_llm"] = (None, "true")
+            
+            # Add max_pages only if it has a value
+            if max_pages is not None:
+                form_data["max_pages"] = (None, str(max_pages))
 
             response = requests.post(
                 self.BASE_MARKER_API_ENDPOINT,

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/docs_to_md/core/processor.py

@@ -53,6 +53,75 @@ class BatchProcessor:
     def should_chunk(self, file_path: Path) -> bool:
         return file_path.suffix.lower() == ".pdf"
 
+    def _chunk_file(
+        self, file_path: Path, tmp_dir: Path, request: ConversionRequest
+    ) -> bool:
+        """Chunk the provided PDF and populate the request object.
+
+        Returns ``True`` if chunking fails and the request should be marked
+        as failed.
+        """
+        try:
+            chunk_result = chunk_pdf_to_temp(str(file_path), self.chunk_size, tmp_dir)
+            if chunk_result:
+                for chunk_info in chunk_result.chunks:
+                    request.add_chunk(Path(chunk_info.path), chunk_info.index)
+                logger.debug(
+                    f"Created {len(request.chunks)} chunks in {tmp_dir}"
+                )
+            else:
+                logger.debug(
+                    f"No chunking needed for {file_path} (<= {self.chunk_size} pages)"
+                )
+            return False
+        except (PDFProcessingError, Exception) as e:
+            logger.error(f"Error chunking PDF {file_path}: {e}", exc_info=True)
+            request.set_status(Status.FAILED, f"Error chunking PDF: {e}")
+            self.cache.save(request)
+            return True
+
+    def _submit_chunks(
+        self, request: ConversionRequest, api_params: ApiParams
+    ) -> bool:
+        """Submit all chunks to the API.
+
+        Returns ``True`` if any submission fails.
+        """
+        submission_failed = False
+        with ProgressTracker(len(request.chunks), "Submitting to API", "chunk") as progress:
+            for chunk in request.ordered_chunks:
+                try:
+                    chunk_request_id = self.client.submit_file(
+                        chunk.path,
+                        output_format=api_params.output_format,
+                        langs=api_params.langs,
+                        use_llm=api_params.use_llm,
+                        strip_existing_ocr=api_params.strip_existing_ocr,
+                        disable_image_extraction=api_params.disable_image_extraction,
+                        force_ocr=api_params.force_ocr,
+                        paginate=api_params.paginate,
+                        max_pages=api_params.max_pages,
+                    )
+                    if chunk_request_id:
+                        chunk.mark_processing(chunk_request_id)
+                    else:
+                        chunk.mark_failed(
+                            f"API submission failed for {chunk.path.name}"
+                        )
+                        submission_failed = True
+                        break
+                except Exception as submit_e:
+                    logger.error(
+                        f"Unexpected error submitting chunk {chunk.path.name}: {submit_e}",
+                        exc_info=True,
+                    )
+                    chunk.mark_failed(f"Error submitting chunk: {submit_e}")
+                    submission_failed = True
+                    break
+                finally:
+                    progress.update()
+        return submission_failed or request.has_failed
+
     def process_file(
         self,
         file_path: Path,
@@ -60,20 +129,11 @@ class BatchProcessor:
         api_params: ApiParams,
         output_paths_obj: OutputPaths,
     ) -> Optional[str]:
-        """
-        Process a single file: chunk if needed, submit to API.
-
-        Args:
-            file_path: Path to the input file.
-            final_output_path: Path where the final output markdown file should be saved.
-            api_params: Parameters for the Marker API call.
-            output_paths_obj: The OutputPaths object containing final markdown and image paths.
+        """Process a single file and submit it to the Marker API.
 
-        Returns:
-            Request ID for tracking if submission is initiated, otherwise None.
-
-        Raises:
-            FileError: Potentially raised by underlying operations (though many are caught).
+        The method now delegates chunking and submission to smaller helpers to
+        keep the logic readable.  It returns the created request ID, which can
+        later be used to poll for results.
         """
         with TemporaryDirectory(self.root_tmp_dir, file_path.stem) as tmp_dir:
             request = ConversionRequest(
@@ -93,30 +153,7 @@ class BatchProcessor:
 
             try:
                 if self.should_chunk(file_path):
-                    try:
-                        chunk_result = chunk_pdf_to_temp(
-                            str(file_path), self.chunk_size, tmp_dir
-                        )
-                        if chunk_result:
-                            for chunk_info in chunk_result.chunks:
-                                request.add_chunk(
-                                    Path(chunk_info.path), chunk_info.index
-                                )
-                            logger.debug(
-                                f"Created {len(request.chunks)} chunks in {tmp_dir}"
-                            )
-                        else:
-                            logger.debug(
-                                f"No chunking needed for {file_path} (<= {self.chunk_size} pages)"
-                            )
-                    except (PDFProcessingError, Exception) as e:
-                        logger.error(
-                            f"Error chunking PDF {file_path}: {e}", exc_info=True
-                        )
-                        request.set_status(
-                            Status.FAILED, f"Error chunking PDF: {str(e)}"
-                        )
-                        self.cache.save(request)
+                    if self._chunk_file(file_path, tmp_dir, request):
                         return request.request_id
 
                 if not request.chunks:
@@ -126,43 +163,9 @@ class BatchProcessor:
                     f"Submitting {len(request.chunks)} chunk(s) to API for {request.original_file.name}..."
                 )
 
-                submission_failed = False
-                with ProgressTracker(
-                    len(request.chunks), "Submitting to API", "chunk"
-                ) as progress:
-                    for chunk in request.ordered_chunks:
-                        try:
-                            chunk_request_id = self.client.submit_file(
-                                chunk.path,
-                                output_format=api_params.output_format,
-                                langs=api_params.langs,
-                                use_llm=api_params.use_llm,
-                                strip_existing_ocr=api_params.strip_existing_ocr,
-                                disable_image_extraction=api_params.disable_image_extraction,
-                                force_ocr=api_params.force_ocr,
-                                paginate=api_params.paginate,
-                                max_pages=api_params.max_pages,
-                            )
-                            if chunk_request_id:
-                                chunk.mark_processing(chunk_request_id)
-                            else:
-                                chunk.mark_failed(
-                                    f"API submission failed for {chunk.path.name}"
-                                )
-                                submission_failed = True
-                                break
-                        except Exception as submit_e:
-                            logger.error(
-                                f"Unexpected error submitting chunk {chunk.path.name}: {submit_e}",
-                                exc_info=True,
-                            )
-                            chunk.mark_failed(f"Error submitting chunk: {submit_e}")
-                            submission_failed = True
-                            break
-                        finally:
-                            progress.update()
-
-                if submission_failed or request.has_failed:
+                submission_failed = self._submit_chunks(request, api_params)
+
+                if submission_failed:
                     request.set_status(
                         Status.FAILED,
                         request.error or "One or more chunk submissions failed.",

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/docs_to_md/core/result_handler.py

@@ -91,9 +91,14 @@ class ResultSaver:
             logger.debug(
                 f"Successfully combined results to {output_file} (Final size: {final_size} bytes)"
             )
-            # Ensure images dir exists before returning (though determine_output_paths should have done it)
-            if req.images_dir:
-                ensure_directory(req.images_dir)
+            # Only create images directory if there are actually images to move
+            if req.images_dir and req.tmp_dir:
+                source_images_dir = req.tmp_dir / "images"
+                if source_images_dir.exists():
+                    ensure_directory(req.images_dir)
+                    logger.debug(f"Created images directory {req.images_dir} (images will be moved)")
+                else:
+                    logger.debug(f"No images found - skipping creation of {req.images_dir}")
             return output_file, final_size
 
         except Exception as e:

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/docs_to_md/storage/models.py

@@ -48,7 +48,8 @@ class ConversionRequest(BaseModel):
     output_format: str = "markdown"
     status: Status = Status.PENDING
     error: Optional[str] = None
-    chunks: List[ChunkInfo] = []
+    # Use default_factory to avoid shared mutable list across instances
+    chunks: List[ChunkInfo] = Field(default_factory=list)
     chunk_size: int
     tmp_dir: Optional[Path] = None  # Directory for temporary files for this conversion
     images_dir: Optional[Path] = None  # Added to store determined image path

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/docs_to_md/utils/file_utils.py

@@ -94,8 +94,9 @@ class FileDiscovery:
             logger.warning(f"Skipping {file_path}: Supported extension '{ext}', but unsupported detected MIME type '{kind.mime}'.")
             return False
         else:
-            logger.info(f"Skipping {file_path}: Supported extension '{ext}', but MIME detection failed or ambiguous.")
-            return False
+            # MIME detection failed - rely on the earlier suffix extension check
+            logger.debug(f"Adding {file_path}: Supported extension '{ext}', MIME detection failed - using extension as fallback.")
+            return True
 
     @staticmethod
     def find_processable_files(

pdf_to_markdown_cli-0.5.1/src/pdf_to_markdown_cli.egg-info/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: pdf-to-markdown-cli
+Version: 0.5.1
+Summary: CLI tool to convert PDF files (and other documents) to markdown using the Marker API.
+Author: Nikita Sokolsky
+License: MIT License
+        
+        Copyright (c) 2025 SokolskyNikita
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/SokolskyNikita/pdf-to-markdown-cli
+Project-URL: Bug Tracker, https://github.com/SokolskyNikita/pdf-to-markdown-cli/issues
+Keywords: pdf,markdown,converter,cli,document,marker,md
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: backoff>=2.0
+Requires-Dist: diskcache>=5.0
+Requires-Dist: filetype>=1.0
+Requires-Dist: pikepdf>=8.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: ratelimit>=2.0
+Requires-Dist: requests>=2.0
+Requires-Dist: tqdm>=4.0
+Provides-Extra: test
+Dynamic: license-file
+
+# PDF to Markdown CLI
+
+Convert PDFs and other documents to Markdown using the [Marker API](https://www.datalab.to/marker).
+
+## Features
+
+- Convert PDFs, Word docs, PowerPoint, spreadsheets, EPUB, HTML, and images to Markdown/JSON/HTML
+- Automatic chunking for large documents with parallel processing
+- Progress tracking and local caching for interrupted runs
+- Full OCR customization options
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install pdf-to-markdown-cli
+```
+
+### From source
+
+```bash
+git clone https://github.com/SokolskyNikita/pdf-to-markdown-cli.git 
+cd pdf-to-markdown-cli
+pip install -e .
+```
+
+## Usage
+
+```bash
+# Get API key from https://www.datalab.to/marker
+export MARKER_PDF_KEY=your_api_key_here
+
+# Basic usage
+pdf-to-md /path/to/file.pdf
+
+# Common options
+pdf-to-md /path/to/file.pdf --json          # JSON output
+pdf-to-md /path/to/file.pdf --noimg         # Disable images  
+pdf-to-md /path/to/file.pdf --max           # Enable all flags for maximum output quality
+```
+
+## CLI Options
+
+- `input`: Input file or directory path
+- `--json`: Output in JSON format (default is markdown)
+- `--langs`: Comma-separated OCR languages (default: "English")
+- `--llm`: Use LLM for enhanced processing
+- `--strip`: Redo OCR processing
+- `--noimg`: Disable image extraction
+- `--force`: Force OCR on all pages
+- `--pages`: Add page delimiters
+- `--max`: Enable all OCR enhancements (equivalent to --llm --strip --force)
+- `-mp`, `--max-pages`: Maximum number of pages to process from the start of the file
+- `--no-chunk`: Disable PDF chunking
+- `-cs`, `--chunk-size`: Set PDF chunk size in pages (default: 25)
+- `-o`, `--output-dir`: Absolute path to the output directory
+- `-v`, `--verbose`: Enable verbose (DEBUG level) logging
+- `--version`: Show the installed version and exit
+
+## Requirements
+
+- Python ≥3.10
+- API key from [datalab.to](https://www.datalab.to/marker)

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/pdf_to_markdown_cli.egg-info/SOURCES.txt

@@ -28,4 +28,9 @@ src/pdf_to_markdown_cli.egg-info/SOURCES.txt
 src/pdf_to_markdown_cli.egg-info/dependency_links.txt
 src/pdf_to_markdown_cli.egg-info/entry_points.txt
 src/pdf_to_markdown_cli.egg-info/requires.txt
-src/pdf_to_markdown_cli.egg-info/top_level.txt
+src/pdf_to_markdown_cli.egg-info/top_level.txt
+tests/test_cli.py
+tests/test_cli_equations.py
+tests/test_paths.py
+tests/test_settings.py
+tests/test_utils.py

{pdf_to_markdown_cli-0.4.0 → pdf_to_markdown_cli-0.5.1}/src/pdf_to_markdown_cli.egg-info/requires.txt

@@ -6,3 +6,5 @@ pydantic>=2.0
 ratelimit>=2.0
 requests>=2.0
 tqdm>=4.0
+
+[test]