visual-parser 2.2.1__tar.gz → 2.2.2__tar.gz

{visual_parser-2.2.1/visual_parser.egg-info → visual_parser-2.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: visual-parser
-Version: 2.2.1
+Version: 2.2.2
 Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
 Author: Zavier N. Ndum
 Author-email: zavier.ndum@tamu.edu
@@ -50,7 +50,7 @@ Dynamic: license-file
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
 
 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
@@ -72,10 +72,11 @@ Dynamic: license-file
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
 
 ## API Keys (.env)
 
@@ -262,6 +263,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -273,8 +275,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)

{visual_parser-2.2.1 → visual_parser-2.2.2}/README.md

@@ -5,7 +5,7 @@
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
 
 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
@@ -27,10 +27,11 @@
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
 
 ## API Keys (.env)
 
@@ -217,6 +218,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -228,8 +230,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)

{visual_parser-2.2.1 → visual_parser-2.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "visual-parser"
-version = "2.2.1"
+version = "2.2.2"
 description = "Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL"
 readme = "README.md"
 requires-python = ">=3.10"

{visual_parser-2.2.1 → visual_parser-2.2.2}/tests/test_config.py

@@ -59,6 +59,20 @@ def test_from_env_reads_redo_names_comma_separated(monkeypatch):
     assert ParserConfig.from_env().redo_names == ["doc1.pdf", "diagram2.png", "doc3.pdf"]
 
 
+def test_nougat_crop_margin_defaults_to_false():
+    assert ParserConfig(input_dir=".").nougat_crop_margin is False
+
+
+def test_from_env_nougat_crop_margin_defaults_to_false(monkeypatch):
+    monkeypatch.delenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", raising=False)
+    assert ParserConfig.from_env().nougat_crop_margin is False
+
+
+def test_from_env_reads_nougat_crop_margin(monkeypatch):
+    monkeypatch.setenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", "true")
+    assert ParserConfig.from_env().nougat_crop_margin is True
+
+
 def test_vision_context_pages_defaults_to_zero(monkeypatch):
     monkeypatch.delenv("VISUAL_PARSER_VISION_CONTEXT_PAGES", raising=False)
     assert ParserConfig.from_env().vision_context_pages == 0

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/__init__.py

@@ -17,4 +17,4 @@ from visual_parser.config import ParserConfig
 from visual_parser.pipeline import run_pipeline
 
 __all__ = ["ParserConfig", "run_pipeline"]
-__version__ = "2.2.1"
+__version__ = "2.2.2"

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/cli.py

@@ -111,6 +111,15 @@ def _build_arg_parser() -> argparse.ArgumentParser:
         metavar="N",
         help="Overlap characters between adjacent chunks (default: 100).",
     )
+    text_group.add_argument(
+        "--nougat-crop-margin",
+        action="store_true",
+        help=(
+            "Let Nougat autocrop whitespace margins (its own default). "
+            "Off by default -- autocropping can silently crop into real "
+            "content on unusual page layouts."
+        ),
+    )
 
     # ---- Vision LLM ---------------------------------------------------------
     vision_group = p.add_argument_group("Vision LLM (figure descriptions & metadata)")
@@ -366,6 +375,7 @@ def main(argv=None) -> int:
         nougat_model          = args.nougat_model,
         chunk_size            = args.chunk_size,
         chunk_overlap         = args.chunk_overlap,
+        nougat_crop_margin    = args.nougat_crop_margin,
         vision_provider       = args.vision_provider,
         gpt_vision_model      = args.vision_model if args.vision_provider == "gpt"    else "gpt-5.4",
         gemini_vision_model   = args.vision_model if args.vision_provider == "gemini" else "gemini-3.8-flash",

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/cli_main.py

@@ -109,6 +109,15 @@ def _build_arg_parser() -> argparse.ArgumentParser:
         metavar="N",
         help="Overlap characters between adjacent chunks (default: 100).",
     )
+    text_group.add_argument(
+        "--nougat-crop-margin",
+        action="store_true",
+        help=(
+            "Let Nougat autocrop whitespace margins (its own default). "
+            "Off by default -- autocropping can silently crop into real "
+            "content on unusual page layouts."
+        ),
+    )
 
     vision_group = parser.add_argument_group("Vision LLM (figure descriptions & metadata)")
     vision_group.add_argument(
@@ -282,6 +291,7 @@ def main(argv=None) -> int:
         nougat_model=args.nougat_model,
         chunk_size=args.chunk_size,
         chunk_overlap=args.chunk_overlap,
+        nougat_crop_margin=args.nougat_crop_margin,
         vision_provider=args.vision_provider,
         gpt_vision_model=args.vision_model if args.vision_provider == "gpt" else "gpt-5.4",
         gemini_vision_model=(

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/config.py

@@ -79,6 +79,14 @@ class ParserConfig:
     nougat_model: str = "facebook/nougat-small"
     """HuggingFace model identifier for Nougat."""
 
+    nougat_crop_margin: bool = False
+    """
+    Passed through to Nougat's image processor as do_crop_margin. False
+    (default) matches this package's long-standing behavior -- autocropping
+    whitespace margins can silently crop into real content on unusual page
+    layouts. Set True to opt into Nougat's own autocrop behavior instead.
+    """
+
     chunk_size: int = 500
     """Target character count per text chunk."""
 
@@ -186,6 +194,7 @@ class ParserConfig:
             output_dir           = os.getenv("VISUAL_PARSER_OUTPUT_DIR", ""),
             text_mode            = os.getenv("VISUAL_PARSER_TEXT_MODE", "nougat"),           # type: ignore[arg-type]
             nougat_model         = os.getenv("VISUAL_PARSER_NOUGAT_MODEL", "facebook/nougat-small"),
+            nougat_crop_margin   = os.getenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", "false").lower() == "true",
             chunk_size           = int(os.getenv("VISUAL_PARSER_CHUNK_SIZE", "500")),
             chunk_overlap        = int(os.getenv("VISUAL_PARSER_CHUNK_OVERLAP", "100")),
             vision_provider      = os.getenv("VISUAL_PARSER_VISION_PROVIDER", "gpt"),        # type: ignore[arg-type]

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/pipeline.py

@@ -501,6 +501,7 @@ def run_pipeline(config: Optional[ParserConfig] = None) -> Dict:
                     chunk_size         = config.chunk_size,
                     chunk_overlap      = config.chunk_overlap,
                     max_workers        = config.max_workers,
+                    nougat_crop_margin = config.nougat_crop_margin,
                 )
                 print(nougat_summary)
             except Exception as exc:

{visual_parser-2.2.1 → visual_parser-2.2.2}/visual_parser/text_extractor.py

@@ -69,6 +69,7 @@ def nougat_extract_pdfs(
     chunk_size: int = 500,
     chunk_overlap: int = 100,
     max_workers: int = 4,
+    nougat_crop_margin: bool = False,
 ) -> Tuple[str, List[str], List[str], int]:
     """
     Extract text from each PDF in *only_process_these* using the Nougat model,
@@ -83,6 +84,9 @@ def nougat_extract_pdfs(
         chunk_size:         Characters per chunk.
         chunk_overlap:      Overlap between adjacent chunks.
         max_workers:        Thread-pool size for parallel PDF processing.
+        nougat_crop_margin: Passed through to Nougat's image processor as
+                            do_crop_margin (default False -- see
+                            ParserConfig.nougat_crop_margin's docstring).
 
     Always resumes against the 04_processed_pdfs.txt registry -- callers
     that want a PDF reprocessed must first clear its existing chunks and
@@ -124,7 +128,7 @@ def nougat_extract_pdfs(
                     pixel_values = processor(
                         images=image,
                         return_tensors="pt",
-                        do_crop_margin=False,
+                        do_crop_margin=nougat_crop_margin,
                     ).pixel_values.to(device)
                 except TypeError:
                     pixel_values = processor(

{visual_parser-2.2.1 → visual_parser-2.2.2/visual_parser.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: visual-parser
-Version: 2.2.1
+Version: 2.2.2
 Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
 Author: Zavier N. Ndum
 Author-email: zavier.ndum@tamu.edu
@@ -50,7 +50,7 @@ Dynamic: license-file
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
 
 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
@@ -72,10 +72,11 @@ Dynamic: license-file
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
 
 ## API Keys (.env)
 
@@ -262,6 +263,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -273,8 +275,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)