visual-parser 2.2.1__tar.gz → 2.2.3__tar.gz

{visual_parser-2.2.1/visual_parser.egg-info → visual_parser-2.2.3}/PKG-INFO

@@ -1,14 +1,14 @@
 Metadata-Version: 2.4
 Name: visual-parser
-Version: 2.2.1
-Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
+Version: 2.2.3
+Summary: Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL
 Author: Zavier N. Ndum
 Author-email: zavier.ndum@tamu.edu
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/SmartLabNuclear/RADIANT_LLM
 Project-URL: Repository, https://github.com/SmartLabNuclear/RADIANT_LLM
 Project-URL: Docker Hub, https://hub.docker.com/r/zev94/radiant-llm
-Keywords: pdf,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
+Keywords: pdf,image,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
@@ -43,16 +43,16 @@ Requires-Dist: ruff; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Dynamic: license-file
 
-# visual-parser (Standalone Visual-RAG PDF Ingestion)
+# visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
 
 ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
 ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
 
-1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
+1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
 
 ## Table of Contents
@@ -72,10 +72,13 @@ Dynamic: license-file
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+
+> **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
 
 ## API Keys (.env)
 
@@ -262,6 +265,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -273,8 +277,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)

{visual_parser-2.2.1 → visual_parser-2.2.3}/README.md

@@ -1,13 +1,13 @@
-# visual-parser (Standalone Visual-RAG PDF Ingestion)
+# visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
 
 ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
 ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
 
-1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
+1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
 
 ## Table of Contents
@@ -27,10 +27,13 @@
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+
+> **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
 
 ## API Keys (.env)
 
@@ -217,6 +220,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -228,8 +232,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)

{visual_parser-2.2.1 → visual_parser-2.2.3}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "visual-parser"
-version = "2.2.1"
-description = "Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL"
+version = "2.2.3"
+description = "Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL"
 readme = "README.md"
 requires-python = ">=3.10"
 license = "Apache-2.0"
@@ -17,7 +17,7 @@ authors = [
 ]
 
 keywords = [
-    "pdf", "rag", "nougat", "vision-llm", "ocr",
+    "pdf", "image", "rag", "nougat", "vision-llm", "ocr",
     "document-parsing", "jsonl", "knowledge-base",
 ]
 

{visual_parser-2.2.1 → visual_parser-2.2.3}/tests/test_config.py

@@ -59,6 +59,20 @@ def test_from_env_reads_redo_names_comma_separated(monkeypatch):
     assert ParserConfig.from_env().redo_names == ["doc1.pdf", "diagram2.png", "doc3.pdf"]
 
 
+def test_nougat_crop_margin_defaults_to_false():
+    assert ParserConfig(input_dir=".").nougat_crop_margin is False
+
+
+def test_from_env_nougat_crop_margin_defaults_to_false(monkeypatch):
+    monkeypatch.delenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", raising=False)
+    assert ParserConfig.from_env().nougat_crop_margin is False
+
+
+def test_from_env_reads_nougat_crop_margin(monkeypatch):
+    monkeypatch.setenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", "true")
+    assert ParserConfig.from_env().nougat_crop_margin is True
+
+
 def test_vision_context_pages_defaults_to_zero(monkeypatch):
     monkeypatch.delenv("VISUAL_PARSER_VISION_CONTEXT_PAGES", raising=False)
     assert ParserConfig.from_env().vision_context_pages == 0

{visual_parser-2.2.1 → visual_parser-2.2.3}/tests/test_image_describer.py

@@ -145,7 +145,7 @@ def test_describe_images_skips_non_string_description(tmp_path, monkeypatch):
     )
 
     assert written == []
-    assert _read_jsonl(tmp_path / "image_descriptions.jsonl") == []
+    assert _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE) == []
 
 
 def test_describe_images_writes_rows_for_each_image(tmp_path, monkeypatch):
@@ -166,7 +166,7 @@ def test_describe_images_writes_rows_for_each_image(tmp_path, monkeypatch):
     )
 
     assert set(written) == {"a.png", "b.png"}
-    rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
+    rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
     assert {r["source"] for r in rows} == {"a.png", "b.png"}
     for row in rows:
         assert row["title"] == "Something"
@@ -195,7 +195,7 @@ def test_failed_image_is_not_marked_written(tmp_path, monkeypatch):
     )
 
     assert written == ["good.png"]
-    rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
+    rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
     assert {r["source"] for r in rows} == {"good.png"}
 
 
@@ -223,7 +223,7 @@ def test_vision_call_exception_is_not_marked_written(tmp_path, monkeypatch):
     )
 
     assert len(written) == 1
-    rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
+    rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
     assert len(rows) == 1
 
 
@@ -231,7 +231,7 @@ def test_resume_skips_already_done_images(tmp_path, monkeypatch):
     img1 = _make_png(tmp_path / "a.png")
     img2 = _make_png(tmp_path / "b.png")
 
-    descriptions_path = tmp_path / "image_descriptions.jsonl"
+    descriptions_path = tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE
     descriptions_path.write_text(
         json.dumps({"source": "a.png", "image_id": "x", "title": "Old", "description": "already done"}) + "\n",
         encoding="utf-8",

{visual_parser-2.2.1 → visual_parser-2.2.3}/tests/test_prompts.py

@@ -14,7 +14,7 @@ def test_figure_prompt_instructs_verbatim_table_transcription():
 def test_figure_prompt_and_image_prompt_use_matching_discrepancy_terminology():
     """Regression test: the two prompts' final heading must use the same
     label and boilerplate 'nothing found' phrase, so the two JSONL outputs
-    (02_visuals_kb.jsonl / image_descriptions.jsonl) don't drift into
+    (02_visuals_kb.jsonl / image_descriptions_kb.jsonl) don't drift into
     inconsistent wording for the same underlying concept."""
     assert "**Discrepancy Check**" in FIGURE_PROMPT
     assert "No discrepancies detected" in FIGURE_PROMPT

visual_parser-2.2.3/visual_parser/__init__.py

@@ -0,0 +1,22 @@
+"""
+visual_parser — Standalone Visual-RAG PDF & Image Parser
+=========================================================
+Detects new PDFs (and standalone images alongside them) in a user-supplied
+directory, extracts text (via Nougat or lightweight PyMuPDF/PyPDFLoader),
+describes every figure/chart/schematic and every standalone image using a
+Vision LLM (OpenAI GPT, Google Gemini, or a local model via Ollama), and
+writes JSONL knowledge bases ready for any downstream RAG system:
+
+    01_chunks_kb.jsonl          – text chunks with stable IDs
+    02_visuals_kb.jsonl         – per-figure visual descriptions
+    03_metadata_kb.jsonl        – document-level metadata (title, authors, DOI …)
+    image_descriptions_kb.jsonl – one holistic description per standalone image
+
+No chatbot, no vector store, no retrieval – just a robust parser.
+"""
+
+from visual_parser.config import ParserConfig
+from visual_parser.pipeline import run_pipeline
+
+__all__ = ["ParserConfig", "run_pipeline"]
+__version__ = "2.2.3"

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/cli.py

@@ -44,13 +44,13 @@ def _build_arg_parser() -> argparse.ArgumentParser:
     p = argparse.ArgumentParser(
         prog="visual-parser",
         description=(
-            "Visual-RAG PDF Parser — detects new PDFs and standalone images, "
+            "Visual-RAG PDF & Image Parser — detects new PDFs and standalone images, "
             "extracts text and figure/image descriptions, and writes JSONL "
             "knowledge bases:\n"
-            "  01_chunks_kb.jsonl       text chunks\n"
-            "  02_visuals_kb.jsonl      PDF figure descriptions\n"
-            "  03_metadata_kb.jsonl     document metadata\n"
-            "  image_descriptions.jsonl standalone image descriptions"
+            "  01_chunks_kb.jsonl            text chunks\n"
+            "  02_visuals_kb.jsonl           PDF figure descriptions\n"
+            "  03_metadata_kb.jsonl          document metadata\n"
+            "  image_descriptions_kb.jsonl   standalone image descriptions"
         ),
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog=USAGE_EXAMPLES,
@@ -111,6 +111,15 @@ def _build_arg_parser() -> argparse.ArgumentParser:
         metavar="N",
         help="Overlap characters between adjacent chunks (default: 100).",
     )
+    text_group.add_argument(
+        "--nougat-crop-margin",
+        action="store_true",
+        help=(
+            "Let Nougat autocrop whitespace margins (its own default). "
+            "Off by default -- autocropping can silently crop into real "
+            "content on unusual page layouts."
+        ),
+    )
 
     # ---- Vision LLM ---------------------------------------------------------
     vision_group = p.add_argument_group("Vision LLM (figure descriptions & metadata)")
@@ -366,6 +375,7 @@ def main(argv=None) -> int:
         nougat_model          = args.nougat_model,
         chunk_size            = args.chunk_size,
         chunk_overlap         = args.chunk_overlap,
+        nougat_crop_margin    = args.nougat_crop_margin,
         vision_provider       = args.vision_provider,
         gpt_vision_model      = args.vision_model if args.vision_provider == "gpt"    else "gpt-5.4",
         gemini_vision_model   = args.vision_model if args.vision_provider == "gemini" else "gemini-3.8-flash",

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/cli_main.py

@@ -1,5 +1,5 @@
 """
-cli_main.py - ASCII-safe CLI entry point for the Visual-RAG PDF Parser.
+cli_main.py - ASCII-safe CLI entry point for the Visual-RAG PDF & Image Parser.
 
 This module exists to keep Windows console help output stable for the
 installed ``visual-parser`` command and ``python -m visual_parser``.
@@ -42,13 +42,13 @@ def _build_arg_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="visual-parser",
         description=(
-            "Visual-RAG PDF Parser - detects new PDFs and standalone images, "
+            "Visual-RAG PDF & Image Parser - detects new PDFs and standalone images, "
             "extracts text and figure/image descriptions, and writes JSONL "
             "knowledge bases:\n"
-            "  01_chunks_kb.jsonl       text chunks\n"
-            "  02_visuals_kb.jsonl      PDF figure descriptions\n"
-            "  03_metadata_kb.jsonl     document metadata\n"
-            "  image_descriptions.jsonl standalone image descriptions"
+            "  01_chunks_kb.jsonl            text chunks\n"
+            "  02_visuals_kb.jsonl           PDF figure descriptions\n"
+            "  03_metadata_kb.jsonl          document metadata\n"
+            "  image_descriptions_kb.jsonl   standalone image descriptions"
         ),
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog=USAGE_EXAMPLES,
@@ -109,6 +109,15 @@ def _build_arg_parser() -> argparse.ArgumentParser:
         metavar="N",
         help="Overlap characters between adjacent chunks (default: 100).",
     )
+    text_group.add_argument(
+        "--nougat-crop-margin",
+        action="store_true",
+        help=(
+            "Let Nougat autocrop whitespace margins (its own default). "
+            "Off by default -- autocropping can silently crop into real "
+            "content on unusual page layouts."
+        ),
+    )
 
     vision_group = parser.add_argument_group("Vision LLM (figure descriptions & metadata)")
     vision_group.add_argument(
@@ -282,6 +291,7 @@ def main(argv=None) -> int:
         nougat_model=args.nougat_model,
         chunk_size=args.chunk_size,
         chunk_overlap=args.chunk_overlap,
+        nougat_crop_margin=args.nougat_crop_margin,
         vision_provider=args.vision_provider,
         gpt_vision_model=args.vision_model if args.vision_provider == "gpt" else "gpt-5.4",
         gemini_vision_model=(

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/config.py

@@ -79,6 +79,14 @@ class ParserConfig:
     nougat_model: str = "facebook/nougat-small"
     """HuggingFace model identifier for Nougat."""
 
+    nougat_crop_margin: bool = False
+    """
+    Passed through to Nougat's image processor as do_crop_margin. False
+    (default) matches this package's long-standing behavior -- autocropping
+    whitespace margins can silently crop into real content on unusual page
+    layouts. Set True to opt into Nougat's own autocrop behavior instead.
+    """
+
     chunk_size: int = 500
     """Target character count per text chunk."""
 
@@ -186,6 +194,7 @@ class ParserConfig:
             output_dir           = os.getenv("VISUAL_PARSER_OUTPUT_DIR", ""),
             text_mode            = os.getenv("VISUAL_PARSER_TEXT_MODE", "nougat"),           # type: ignore[arg-type]
             nougat_model         = os.getenv("VISUAL_PARSER_NOUGAT_MODEL", "facebook/nougat-small"),
+            nougat_crop_margin   = os.getenv("VISUAL_PARSER_NOUGAT_CROP_MARGIN", "false").lower() == "true",
             chunk_size           = int(os.getenv("VISUAL_PARSER_CHUNK_SIZE", "500")),
             chunk_overlap        = int(os.getenv("VISUAL_PARSER_CHUNK_OVERLAP", "100")),
             vision_provider      = os.getenv("VISUAL_PARSER_VISION_PROVIDER", "gpt"),        # type: ignore[arg-type]

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/image_describer.py

@@ -10,7 +10,7 @@ supplied image files instead of PyMuPDF-rasterized PDF pages.
 
 Output
 ------
-One record per image is appended to ``image_descriptions.jsonl`` in
+One record per image is appended to ``image_descriptions_kb.jsonl`` in
 *output_dir*:
 
     {
@@ -40,7 +40,7 @@ from visual_parser.vision_llm import call_vision_llm
 
 logger = logging.getLogger(__name__)
 
-IMAGE_DESCRIPTIONS_FILE = "image_descriptions.jsonl"
+IMAGE_DESCRIPTIONS_FILE = "image_descriptions_kb.jsonl"
 
 # Decoupled from figure_describer.py's context-window-scaled formula (10 min
 # for 1 image there): IMAGE_DESCRIPTION_PROMPT's "Verbatim Text Content"
@@ -171,7 +171,7 @@ def describe_images(
     """
     For each image in *new_image_paths*, call the Vision LLM once (no
     context window -- a single standalone image per call), parse the
-    description, and append the result to ``image_descriptions.jsonl`` in
+    description, and append the result to ``image_descriptions_kb.jsonl`` in
     *output_dir*.
 
     Returns the basenames actually written a row for -- NOT the full input
@@ -183,7 +183,7 @@ def describe_images(
 
     Args:
         new_image_paths:  Full paths of images to describe.
-        output_dir:       Directory where ``image_descriptions.jsonl`` is written.
+        output_dir:       Directory where ``image_descriptions_kb.jsonl`` is written.
         vision_provider:  ``'gpt'``, ``'gemini'``, or ``'ollama'``.
         vision_api_key:   API key for the chosen provider (unused for ollama).
         vision_model:     Vision model name string.
@@ -191,7 +191,7 @@ def describe_images(
         image_prompt:     The instruction prompt sent with each image.
         max_workers:      Thread-pool size for concurrent vision-LLM calls.
 
-    Always resumes from image_descriptions.jsonl's existing content (via the
+    Always resumes from image_descriptions_kb.jsonl's existing content (via the
     done_images set below) -- callers that want an image redescribed must
     first remove its existing row and its basename from the processed-
     images registry (see kb_redo.redo_entries()), then pass it in here as if

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/kb_redo.py

@@ -14,7 +14,7 @@ dependency-free package and cannot depend on the sibling RADIANT-LLM repo)
 the proven basename-match + backup-once + atomic-rewrite pattern from
 RADIANT-LLM's knowledge_base_sanitizer() (radiant_lmm/RADIANT_LLM_ChatBot_
 Local/utils/pdf_helpers.py:1464-1607), extended to the 4th JSONL file
-(image_descriptions.jsonl) and both registries (04_processed_pdfs.txt,
+(image_descriptions_kb.jsonl) and both registries (04_processed_pdfs.txt,
 parsed_images.txt), which that function never touches at all.
 """
 
@@ -25,6 +25,7 @@ import os
 import shutil
 from typing import Any, Dict, Iterable, List
 
+from visual_parser.image_describer import IMAGE_DESCRIPTIONS_FILE
 from visual_parser.image_tracker import PROCESSED_IMAGES_REGISTRY
 from visual_parser.jsonl_writer import atomic_rewrite_jsonl
 from visual_parser.pdf_tracker import PROCESSED_REGISTRY, load_processed_pdfs, unmark_as_processed
@@ -33,12 +34,16 @@ logger = logging.getLogger(__name__)
 
 # Hardcoded, not glob("*.jsonl") like the RADIANT precedent -- output_dir
 # often equals input_dir (the documented default), so globbing risks
-# touching unrelated JSONL files a user happens to keep there.
+# touching unrelated JSONL files a user happens to keep there. The image
+# filename is imported (not hardcoded like the other three) because it was
+# renamed once already (image_descriptions.jsonl -> image_descriptions_kb.jsonl,
+# 2.2.3) with this exact file as a hardcoded duplicate that needed a manual
+# fix -- importing the constant means it can't drift out of sync again.
 KB_JSONL_FILENAMES = (
     "01_chunks_kb.jsonl",
     "02_visuals_kb.jsonl",
     "03_metadata_kb.jsonl",
-    "image_descriptions.jsonl",
+    IMAGE_DESCRIPTIONS_FILE,
 )
 REGISTRY_FILENAMES = (PROCESSED_REGISTRY, PROCESSED_IMAGES_REGISTRY)
 BACKUP_SUFFIX = "_backup_before_redo"

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/pipeline.py

@@ -501,6 +501,7 @@ def run_pipeline(config: Optional[ParserConfig] = None) -> Dict:
                     chunk_size         = config.chunk_size,
                     chunk_overlap      = config.chunk_overlap,
                     max_workers        = config.max_workers,
+                    nougat_crop_margin = config.nougat_crop_margin,
                 )
                 print(nougat_summary)
             except Exception as exc:

{visual_parser-2.2.1 → visual_parser-2.2.3}/visual_parser/text_extractor.py

@@ -69,6 +69,7 @@ def nougat_extract_pdfs(
     chunk_size: int = 500,
     chunk_overlap: int = 100,
     max_workers: int = 4,
+    nougat_crop_margin: bool = False,
 ) -> Tuple[str, List[str], List[str], int]:
     """
     Extract text from each PDF in *only_process_these* using the Nougat model,
@@ -83,6 +84,9 @@ def nougat_extract_pdfs(
         chunk_size:         Characters per chunk.
         chunk_overlap:      Overlap between adjacent chunks.
         max_workers:        Thread-pool size for parallel PDF processing.
+        nougat_crop_margin: Passed through to Nougat's image processor as
+                            do_crop_margin (default False -- see
+                            ParserConfig.nougat_crop_margin's docstring).
 
     Always resumes against the 04_processed_pdfs.txt registry -- callers
     that want a PDF reprocessed must first clear its existing chunks and
@@ -124,7 +128,7 @@ def nougat_extract_pdfs(
                     pixel_values = processor(
                         images=image,
                         return_tensors="pt",
-                        do_crop_margin=False,
+                        do_crop_margin=nougat_crop_margin,
                     ).pixel_values.to(device)
                 except TypeError:
                     pixel_values = processor(

{visual_parser-2.2.1 → visual_parser-2.2.3/visual_parser.egg-info}/PKG-INFO

@@ -1,14 +1,14 @@
 Metadata-Version: 2.4
 Name: visual-parser
-Version: 2.2.1
-Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
+Version: 2.2.3
+Summary: Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL
 Author: Zavier N. Ndum
 Author-email: zavier.ndum@tamu.edu
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/SmartLabNuclear/RADIANT_LLM
 Project-URL: Repository, https://github.com/SmartLabNuclear/RADIANT_LLM
 Project-URL: Docker Hub, https://hub.docker.com/r/zev94/radiant-llm
-Keywords: pdf,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
+Keywords: pdf,image,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
@@ -43,16 +43,16 @@ Requires-Dist: ruff; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Dynamic: license-file
 
-# visual-parser (Standalone Visual-RAG PDF Ingestion)
+# visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
 
 ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
 ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
 ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
 ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
 
-`visual-parser` converts PDFs into a multi-modal JSONL knowledge base (text chunks, figure descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
+`visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
 
-1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
+1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
 2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
 
 ## Table of Contents
@@ -72,10 +72,13 @@ Dynamic: license-file
 
 By default, the pipeline writes:
 - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
-- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM).
+- `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
 - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
+- `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
 
-Alongside the KB, the pipeline also writes two bookkeeping files: `04_processed_pdfs.txt` (tracks which PDFs have already been processed, so re-runs skip them unless `--rebuild`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
+
+> **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
 
 ## API Keys (.env)
 
@@ -262,6 +265,7 @@ Text extraction:
 - `--nougat-model facebook/nougat-small`
 - `--chunk-size 500`
 - `--chunk-overlap 100`
+- `--nougat-crop-margin` (off by default — lets Nougat autocrop whitespace margins instead of this package's long-standing uncropped behavior; autocropping can silently crop into real content on unusual page layouts)
 
 Vision LLM:
 - `--vision-provider gpt|gemini|ollama` (default: `gpt`) — see [Local Ollama vision provider](#local-ollama-vision-provider) for the `ollama` option
@@ -273,8 +277,10 @@ Vision LLM:
 
 Performance / misc:
 - `--max-workers 4`
-- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`)
+- `--rebuild` (reprocess everything; ignore `04_processed_pdfs.txt`/`parsed_images.txt`)
+- `--redo NAME` (surgically clear one PDF/image's rows from every KB file + both registries, so it's reprocessed fresh on the next run — a targeted alternative to `--rebuild`)
 - `--skip-text` (skip text extraction and resume only the vision steps; use after an interrupted run)
+- `--skip-images` (don't process standalone images in `--input-dir`, PDFs only)
 - `--list-models` — print the live, currently-available vision models for whichever provider(s) you have a key configured for, then exit (no PDF processing). Reflects Portkey's catalog too when that's active.
 - `--version` / `-V`
 - `--log-level DEBUG|INFO|WARNING|ERROR` (default: `ERROR`)

visual_parser-2.2.1/visual_parser/__init__.py

@@ -1,20 +0,0 @@
-"""
-visual_parser — Standalone Visual-RAG PDF Parser
-=================================================
-Detects new PDFs in a user-supplied directory, extracts text (via Nougat or
-lightweight PyMuPDF/PyPDFLoader), describes every figure/chart/schematic using
-a Vision LLM (OpenAI GPT, Google Gemini, or a local model via Ollama), and
-writes three JSONL knowledge bases ready for any downstream RAG system:
-
-    01_chunks_kb.jsonl   – text chunks with stable IDs
-    02_visuals_kb.jsonl  – per-figure visual descriptions
-    03_metadata_kb.jsonl – document-level metadata (title, authors, DOI …)
-
-No chatbot, no vector store, no retrieval – just a robust parser.
-"""
-
-from visual_parser.config import ParserConfig
-from visual_parser.pipeline import run_pipeline
-
-__all__ = ["ParserConfig", "run_pipeline"]
-__version__ = "2.2.1"