visual-parser 2.2.2__tar.gz → 2.2.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. {visual_parser-2.2.2/visual_parser.egg-info → visual_parser-2.2.3}/PKG-INFO +9 -7
  2. {visual_parser-2.2.2 → visual_parser-2.2.3}/README.md +6 -4
  3. {visual_parser-2.2.2 → visual_parser-2.2.3}/pyproject.toml +3 -3
  4. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_image_describer.py +5 -5
  5. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_prompts.py +1 -1
  6. visual_parser-2.2.3/visual_parser/__init__.py +22 -0
  7. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/cli.py +5 -5
  8. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/cli_main.py +6 -6
  9. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/image_describer.py +5 -5
  10. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/kb_redo.py +8 -3
  11. {visual_parser-2.2.2 → visual_parser-2.2.3/visual_parser.egg-info}/PKG-INFO +9 -7
  12. visual_parser-2.2.2/visual_parser/__init__.py +0 -20
  13. {visual_parser-2.2.2 → visual_parser-2.2.3}/LICENSE +0 -0
  14. {visual_parser-2.2.2 → visual_parser-2.2.3}/setup.cfg +0 -0
  15. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_cli_parity.py +0 -0
  16. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_config.py +0 -0
  17. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_figure_describer.py +0 -0
  18. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_image_tracker.py +0 -0
  19. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_integration.py +0 -0
  20. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_jsonl_writer.py +0 -0
  21. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_kb_redo.py +0 -0
  22. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_metadata_extractor.py +0 -0
  23. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_model_catalog.py +0 -0
  24. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_ollama_local.py +0 -0
  25. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_pdf_tracker.py +0 -0
  26. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_pipeline.py +0 -0
  27. {visual_parser-2.2.2 → visual_parser-2.2.3}/tests/test_vision_llm.py +0 -0
  28. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/__main__.py +0 -0
  29. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/config.py +0 -0
  30. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/figure_describer.py +0 -0
  31. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/image_tracker.py +0 -0
  32. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/jsonl_writer.py +0 -0
  33. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/metadata_extractor.py +0 -0
  34. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/model_catalog.py +0 -0
  35. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/nougat_engine.py +0 -0
  36. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/ollama_local.py +0 -0
  37. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/openai_gateway.py +0 -0
  38. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/pdf_tracker.py +0 -0
  39. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/pipeline.py +0 -0
  40. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/prompts.py +0 -0
  41. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/text_extractor.py +0 -0
  42. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser/vision_llm.py +0 -0
  43. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser.egg-info/SOURCES.txt +0 -0
  44. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser.egg-info/dependency_links.txt +0 -0
  45. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser.egg-info/entry_points.txt +0 -0
  46. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser.egg-info/requires.txt +0 -0
  47. {visual_parser-2.2.2 → visual_parser-2.2.3}/visual_parser.egg-info/top_level.txt +0 -0
@@ -1,14 +1,14 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: visual-parser
3
- Version: 2.2.2
4
- Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
3
+ Version: 2.2.3
4
+ Summary: Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL
5
5
  Author: Zavier N. Ndum
6
6
  Author-email: zavier.ndum@tamu.edu
7
7
  License-Expression: Apache-2.0
8
8
  Project-URL: Homepage, https://github.com/SmartLabNuclear/RADIANT_LLM
9
9
  Project-URL: Repository, https://github.com/SmartLabNuclear/RADIANT_LLM
10
10
  Project-URL: Docker Hub, https://hub.docker.com/r/zev94/radiant-llm
11
- Keywords: pdf,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
11
+ Keywords: pdf,image,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
12
12
  Classifier: Programming Language :: Python :: 3
13
13
  Classifier: Programming Language :: Python :: 3.10
14
14
  Classifier: Programming Language :: Python :: 3.11
@@ -43,16 +43,16 @@ Requires-Dist: ruff; extra == "dev"
43
43
  Requires-Dist: mypy; extra == "dev"
44
44
  Dynamic: license-file
45
45
 
46
- # visual-parser (Standalone Visual-RAG PDF Ingestion)
46
+ # visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
47
47
 
48
48
  ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
49
49
  ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
50
50
  ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
51
51
  ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
52
52
 
53
- `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
53
+ `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
54
54
 
55
- 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
55
+ 1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
56
56
  2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
57
57
 
58
58
  ## Table of Contents
@@ -74,10 +74,12 @@ By default, the pipeline writes:
74
74
  - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
75
75
  - `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
76
76
  - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
77
- - `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
77
+ - `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
78
78
 
79
79
  Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
80
80
 
81
+ > **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
82
+
81
83
  ## API Keys (.env)
82
84
 
83
85
  Provide at least one provider:
@@ -1,13 +1,13 @@
1
- # visual-parser (Standalone Visual-RAG PDF Ingestion)
1
+ # visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
2
2
 
3
3
  ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
4
4
  ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
5
5
  ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
6
6
  ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
7
7
 
8
- `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
8
+ `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
9
9
 
10
- 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
10
+ 1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
11
11
  2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
12
12
 
13
13
  ## Table of Contents
@@ -29,10 +29,12 @@ By default, the pipeline writes:
29
29
  - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
30
30
  - `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
31
31
  - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
32
- - `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
32
+ - `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
33
33
 
34
34
  Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
35
35
 
36
+ > **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
37
+
36
38
  ## API Keys (.env)
37
39
 
38
40
  Provide at least one provider:
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "visual-parser"
7
- version = "2.2.2"
8
- description = "Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL"
7
+ version = "2.2.3"
8
+ description = "Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
11
11
  license = "Apache-2.0"
@@ -17,7 +17,7 @@ authors = [
17
17
  ]
18
18
 
19
19
  keywords = [
20
- "pdf", "rag", "nougat", "vision-llm", "ocr",
20
+ "pdf", "image", "rag", "nougat", "vision-llm", "ocr",
21
21
  "document-parsing", "jsonl", "knowledge-base",
22
22
  ]
23
23
 
@@ -145,7 +145,7 @@ def test_describe_images_skips_non_string_description(tmp_path, monkeypatch):
145
145
  )
146
146
 
147
147
  assert written == []
148
- assert _read_jsonl(tmp_path / "image_descriptions.jsonl") == []
148
+ assert _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE) == []
149
149
 
150
150
 
151
151
  def test_describe_images_writes_rows_for_each_image(tmp_path, monkeypatch):
@@ -166,7 +166,7 @@ def test_describe_images_writes_rows_for_each_image(tmp_path, monkeypatch):
166
166
  )
167
167
 
168
168
  assert set(written) == {"a.png", "b.png"}
169
- rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
169
+ rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
170
170
  assert {r["source"] for r in rows} == {"a.png", "b.png"}
171
171
  for row in rows:
172
172
  assert row["title"] == "Something"
@@ -195,7 +195,7 @@ def test_failed_image_is_not_marked_written(tmp_path, monkeypatch):
195
195
  )
196
196
 
197
197
  assert written == ["good.png"]
198
- rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
198
+ rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
199
199
  assert {r["source"] for r in rows} == {"good.png"}
200
200
 
201
201
 
@@ -223,7 +223,7 @@ def test_vision_call_exception_is_not_marked_written(tmp_path, monkeypatch):
223
223
  )
224
224
 
225
225
  assert len(written) == 1
226
- rows = _read_jsonl(tmp_path / "image_descriptions.jsonl")
226
+ rows = _read_jsonl(tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE)
227
227
  assert len(rows) == 1
228
228
 
229
229
 
@@ -231,7 +231,7 @@ def test_resume_skips_already_done_images(tmp_path, monkeypatch):
231
231
  img1 = _make_png(tmp_path / "a.png")
232
232
  img2 = _make_png(tmp_path / "b.png")
233
233
 
234
- descriptions_path = tmp_path / "image_descriptions.jsonl"
234
+ descriptions_path = tmp_path / image_describer.IMAGE_DESCRIPTIONS_FILE
235
235
  descriptions_path.write_text(
236
236
  json.dumps({"source": "a.png", "image_id": "x", "title": "Old", "description": "already done"}) + "\n",
237
237
  encoding="utf-8",
@@ -14,7 +14,7 @@ def test_figure_prompt_instructs_verbatim_table_transcription():
14
14
  def test_figure_prompt_and_image_prompt_use_matching_discrepancy_terminology():
15
15
  """Regression test: the two prompts' final heading must use the same
16
16
  label and boilerplate 'nothing found' phrase, so the two JSONL outputs
17
- (02_visuals_kb.jsonl / image_descriptions.jsonl) don't drift into
17
+ (02_visuals_kb.jsonl / image_descriptions_kb.jsonl) don't drift into
18
18
  inconsistent wording for the same underlying concept."""
19
19
  assert "**Discrepancy Check**" in FIGURE_PROMPT
20
20
  assert "No discrepancies detected" in FIGURE_PROMPT
@@ -0,0 +1,22 @@
1
+ """
2
+ visual_parser — Standalone Visual-RAG PDF & Image Parser
3
+ =========================================================
4
+ Detects new PDFs (and standalone images alongside them) in a user-supplied
5
+ directory, extracts text (via Nougat or lightweight PyMuPDF/PyPDFLoader),
6
+ describes every figure/chart/schematic and every standalone image using a
7
+ Vision LLM (OpenAI GPT, Google Gemini, or a local model via Ollama), and
8
+ writes JSONL knowledge bases ready for any downstream RAG system:
9
+
10
+ 01_chunks_kb.jsonl – text chunks with stable IDs
11
+ 02_visuals_kb.jsonl – per-figure visual descriptions
12
+ 03_metadata_kb.jsonl – document-level metadata (title, authors, DOI …)
13
+ image_descriptions_kb.jsonl – one holistic description per standalone image
14
+
15
+ No chatbot, no vector store, no retrieval – just a robust parser.
16
+ """
17
+
18
+ from visual_parser.config import ParserConfig
19
+ from visual_parser.pipeline import run_pipeline
20
+
21
+ __all__ = ["ParserConfig", "run_pipeline"]
22
+ __version__ = "2.2.3"
@@ -44,13 +44,13 @@ def _build_arg_parser() -> argparse.ArgumentParser:
44
44
  p = argparse.ArgumentParser(
45
45
  prog="visual-parser",
46
46
  description=(
47
- "Visual-RAG PDF Parser — detects new PDFs and standalone images, "
47
+ "Visual-RAG PDF & Image Parser — detects new PDFs and standalone images, "
48
48
  "extracts text and figure/image descriptions, and writes JSONL "
49
49
  "knowledge bases:\n"
50
- " 01_chunks_kb.jsonl text chunks\n"
51
- " 02_visuals_kb.jsonl PDF figure descriptions\n"
52
- " 03_metadata_kb.jsonl document metadata\n"
53
- " image_descriptions.jsonl standalone image descriptions"
50
+ " 01_chunks_kb.jsonl text chunks\n"
51
+ " 02_visuals_kb.jsonl PDF figure descriptions\n"
52
+ " 03_metadata_kb.jsonl document metadata\n"
53
+ " image_descriptions_kb.jsonl standalone image descriptions"
54
54
  ),
55
55
  formatter_class=argparse.RawDescriptionHelpFormatter,
56
56
  epilog=USAGE_EXAMPLES,
@@ -1,5 +1,5 @@
1
1
  """
2
- cli_main.py - ASCII-safe CLI entry point for the Visual-RAG PDF Parser.
2
+ cli_main.py - ASCII-safe CLI entry point for the Visual-RAG PDF & Image Parser.
3
3
 
4
4
  This module exists to keep Windows console help output stable for the
5
5
  installed ``visual-parser`` command and ``python -m visual_parser``.
@@ -42,13 +42,13 @@ def _build_arg_parser() -> argparse.ArgumentParser:
42
42
  parser = argparse.ArgumentParser(
43
43
  prog="visual-parser",
44
44
  description=(
45
- "Visual-RAG PDF Parser - detects new PDFs and standalone images, "
45
+ "Visual-RAG PDF & Image Parser - detects new PDFs and standalone images, "
46
46
  "extracts text and figure/image descriptions, and writes JSONL "
47
47
  "knowledge bases:\n"
48
- " 01_chunks_kb.jsonl text chunks\n"
49
- " 02_visuals_kb.jsonl PDF figure descriptions\n"
50
- " 03_metadata_kb.jsonl document metadata\n"
51
- " image_descriptions.jsonl standalone image descriptions"
48
+ " 01_chunks_kb.jsonl text chunks\n"
49
+ " 02_visuals_kb.jsonl PDF figure descriptions\n"
50
+ " 03_metadata_kb.jsonl document metadata\n"
51
+ " image_descriptions_kb.jsonl standalone image descriptions"
52
52
  ),
53
53
  formatter_class=argparse.RawDescriptionHelpFormatter,
54
54
  epilog=USAGE_EXAMPLES,
@@ -10,7 +10,7 @@ supplied image files instead of PyMuPDF-rasterized PDF pages.
10
10
 
11
11
  Output
12
12
  ------
13
- One record per image is appended to ``image_descriptions.jsonl`` in
13
+ One record per image is appended to ``image_descriptions_kb.jsonl`` in
14
14
  *output_dir*:
15
15
 
16
16
  {
@@ -40,7 +40,7 @@ from visual_parser.vision_llm import call_vision_llm
40
40
 
41
41
  logger = logging.getLogger(__name__)
42
42
 
43
- IMAGE_DESCRIPTIONS_FILE = "image_descriptions.jsonl"
43
+ IMAGE_DESCRIPTIONS_FILE = "image_descriptions_kb.jsonl"
44
44
 
45
45
  # Decoupled from figure_describer.py's context-window-scaled formula (10 min
46
46
  # for 1 image there): IMAGE_DESCRIPTION_PROMPT's "Verbatim Text Content"
@@ -171,7 +171,7 @@ def describe_images(
171
171
  """
172
172
  For each image in *new_image_paths*, call the Vision LLM once (no
173
173
  context window -- a single standalone image per call), parse the
174
- description, and append the result to ``image_descriptions.jsonl`` in
174
+ description, and append the result to ``image_descriptions_kb.jsonl`` in
175
175
  *output_dir*.
176
176
 
177
177
  Returns the basenames actually written a row for -- NOT the full input
@@ -183,7 +183,7 @@ def describe_images(
183
183
 
184
184
  Args:
185
185
  new_image_paths: Full paths of images to describe.
186
- output_dir: Directory where ``image_descriptions.jsonl`` is written.
186
+ output_dir: Directory where ``image_descriptions_kb.jsonl`` is written.
187
187
  vision_provider: ``'gpt'``, ``'gemini'``, or ``'ollama'``.
188
188
  vision_api_key: API key for the chosen provider (unused for ollama).
189
189
  vision_model: Vision model name string.
@@ -191,7 +191,7 @@ def describe_images(
191
191
  image_prompt: The instruction prompt sent with each image.
192
192
  max_workers: Thread-pool size for concurrent vision-LLM calls.
193
193
 
194
- Always resumes from image_descriptions.jsonl's existing content (via the
194
+ Always resumes from image_descriptions_kb.jsonl's existing content (via the
195
195
  done_images set below) -- callers that want an image redescribed must
196
196
  first remove its existing row and its basename from the processed-
197
197
  images registry (see kb_redo.redo_entries()), then pass it in here as if
@@ -14,7 +14,7 @@ dependency-free package and cannot depend on the sibling RADIANT-LLM repo)
14
14
  the proven basename-match + backup-once + atomic-rewrite pattern from
15
15
  RADIANT-LLM's knowledge_base_sanitizer() (radiant_lmm/RADIANT_LLM_ChatBot_
16
16
  Local/utils/pdf_helpers.py:1464-1607), extended to the 4th JSONL file
17
- (image_descriptions.jsonl) and both registries (04_processed_pdfs.txt,
17
+ (image_descriptions_kb.jsonl) and both registries (04_processed_pdfs.txt,
18
18
  parsed_images.txt), which that function never touches at all.
19
19
  """
20
20
 
@@ -25,6 +25,7 @@ import os
25
25
  import shutil
26
26
  from typing import Any, Dict, Iterable, List
27
27
 
28
+ from visual_parser.image_describer import IMAGE_DESCRIPTIONS_FILE
28
29
  from visual_parser.image_tracker import PROCESSED_IMAGES_REGISTRY
29
30
  from visual_parser.jsonl_writer import atomic_rewrite_jsonl
30
31
  from visual_parser.pdf_tracker import PROCESSED_REGISTRY, load_processed_pdfs, unmark_as_processed
@@ -33,12 +34,16 @@ logger = logging.getLogger(__name__)
33
34
 
34
35
  # Hardcoded, not glob("*.jsonl") like the RADIANT precedent -- output_dir
35
36
  # often equals input_dir (the documented default), so globbing risks
36
- # touching unrelated JSONL files a user happens to keep there.
37
+ # touching unrelated JSONL files a user happens to keep there. The image
38
+ # filename is imported (not hardcoded like the other three) because it was
39
+ # renamed once already (image_descriptions.jsonl -> image_descriptions_kb.jsonl,
40
+ # 2.2.3) with this exact file as a hardcoded duplicate that needed a manual
41
+ # fix -- importing the constant means it can't drift out of sync again.
37
42
  KB_JSONL_FILENAMES = (
38
43
  "01_chunks_kb.jsonl",
39
44
  "02_visuals_kb.jsonl",
40
45
  "03_metadata_kb.jsonl",
41
- "image_descriptions.jsonl",
46
+ IMAGE_DESCRIPTIONS_FILE,
42
47
  )
43
48
  REGISTRY_FILENAMES = (PROCESSED_REGISTRY, PROCESSED_IMAGES_REGISTRY)
44
49
  BACKUP_SUFFIX = "_backup_before_redo"
@@ -1,14 +1,14 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: visual-parser
3
- Version: 2.2.2
4
- Summary: Standalone Visual-RAG PDF Parser - text extraction and Vision-LLM figure descriptions to JSONL
3
+ Version: 2.2.3
4
+ Summary: Standalone Visual-RAG PDF & Image Parser - text extraction and Vision-LLM figure/image descriptions to JSONL
5
5
  Author: Zavier N. Ndum
6
6
  Author-email: zavier.ndum@tamu.edu
7
7
  License-Expression: Apache-2.0
8
8
  Project-URL: Homepage, https://github.com/SmartLabNuclear/RADIANT_LLM
9
9
  Project-URL: Repository, https://github.com/SmartLabNuclear/RADIANT_LLM
10
10
  Project-URL: Docker Hub, https://hub.docker.com/r/zev94/radiant-llm
11
- Keywords: pdf,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
11
+ Keywords: pdf,image,rag,nougat,vision-llm,ocr,document-parsing,jsonl,knowledge-base
12
12
  Classifier: Programming Language :: Python :: 3
13
13
  Classifier: Programming Language :: Python :: 3.10
14
14
  Classifier: Programming Language :: Python :: 3.11
@@ -43,16 +43,16 @@ Requires-Dist: ruff; extra == "dev"
43
43
  Requires-Dist: mypy; extra == "dev"
44
44
  Dynamic: license-file
45
45
 
46
- # visual-parser (Standalone Visual-RAG PDF Ingestion)
46
+ # visual-parser (Standalone Visual-RAG PDF & Image Ingestion)
47
47
 
48
48
  ![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-brightgreen.svg)
49
49
  ![PyTorch](https://img.shields.io/badge/PyTorch-2.1%2B-ee4c2c.svg)
50
50
  ![LangChain](https://img.shields.io/badge/LangChain-0.1%2B-1C3C3C.svg)
51
51
  ![CUDA](https://img.shields.io/badge/CUDA-optional-76B900.svg)
52
52
 
53
- `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-ingestion tool, independent of any chatbot.
53
+ `visual-parser` converts PDFs (and standalone images alongside them) into a multi-modal JSONL knowledge base (text chunks, figure descriptions, standalone image descriptions, metadata). It was extracted from [RADIANT-LLM](https://github.com/SmartLabNuclear/RADIANT_LLM) as a standalone PDF-and-image ingestion tool, independent of any chatbot.
54
54
 
55
- 1) Run `visual-parser` on curated PDFs to generate JSONL KB files.
55
+ 1) Run `visual-parser` on a directory of curated PDFs and/or standalone images to generate JSONL KB files.
56
56
  2) Point any downstream RAG system at the generated KB — RADIANT-LLM, AutoSAM, and AutoFLUKA all consume the identical JSONL/registry format, so the same KB works with any of them without re-parsing.
57
57
 
58
58
  ## Table of Contents
@@ -74,10 +74,12 @@ By default, the pipeline writes:
74
74
  - `01_chunks_kb.jsonl`: chunked text extracted from PDFs (Nougat by default).
75
75
  - `02_visuals_kb.jsonl`: figure/page visual descriptions (Vision LLM), one row per figure found in a PDF.
76
76
  - `03_metadata_kb.jsonl`: document metadata rows (title/author/etc.).
77
- - `image_descriptions.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
77
+ - `image_descriptions_kb.jsonl`: one holistic description per **standalone image** file found alongside your PDFs in `--input-dir` (not a PDF's own figures) — auto-detected by extension, no separate flag needed; disable with `--skip-images`.
78
78
 
79
79
  Alongside the KB, the pipeline also writes bookkeeping files: `04_processed_pdfs.txt` / `parsed_images.txt` (track which PDFs/images have already been processed, so re-runs skip them unless `--rebuild` or `--redo NAME`) and `05_pipeline.log` (run log at the verbosity set by `--log-level`, default `ERROR`).
80
80
 
81
+ > **Breaking change in 2.2.3:** the standalone-image KB file was renamed from `image_descriptions.jsonl` to `image_descriptions_kb.jsonl` (matching the `NN_name_kb.jsonl` convention used by the other three KB files). Upgrading from an older version does **not** migrate existing data automatically — the resume check only looks at the new filename, so already-described images in an old `image_descriptions.jsonl` would otherwise appear "new" again and get re-described. If you have existing data, rename the file yourself before re-running: `image_descriptions.jsonl` → `image_descriptions_kb.jsonl`.
82
+
81
83
  ## API Keys (.env)
82
84
 
83
85
  Provide at least one provider:
@@ -1,20 +0,0 @@
1
- """
2
- visual_parser — Standalone Visual-RAG PDF Parser
3
- =================================================
4
- Detects new PDFs in a user-supplied directory, extracts text (via Nougat or
5
- lightweight PyMuPDF/PyPDFLoader), describes every figure/chart/schematic using
6
- a Vision LLM (OpenAI GPT, Google Gemini, or a local model via Ollama), and
7
- writes three JSONL knowledge bases ready for any downstream RAG system:
8
-
9
- 01_chunks_kb.jsonl – text chunks with stable IDs
10
- 02_visuals_kb.jsonl – per-figure visual descriptions
11
- 03_metadata_kb.jsonl – document-level metadata (title, authors, DOI …)
12
-
13
- No chatbot, no vector store, no retrieval – just a robust parser.
14
- """
15
-
16
- from visual_parser.config import ParserConfig
17
- from visual_parser.pipeline import run_pipeline
18
-
19
- __all__ = ["ParserConfig", "run_pipeline"]
20
- __version__ = "2.2.2"
File without changes
File without changes