nemo-retriever 2026.8.13.dev182__tar.gz → 2026.8.14.dev183__tar.gz

{nemo_retriever-2026.8.13.dev182/src/nemo_retriever.egg-info → nemo_retriever-2026.8.14.dev183}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nemo-retriever
-Version: 2026.8.13.dev182
+Version: 2026.8.14.dev183
 Summary: A modern RAG ingestion pipeline from Nvidia
 Author-email: Jeremy Dyer <jdyer@nvidia.com>
 License-Expression: Apache-2.0
@@ -45,7 +45,7 @@ Requires-Dist: markitdown
 Requires-Dist: langchain-nvidia-ai-endpoints>=1.4.0
 Requires-Dist: lancedb
 Requires-Dist: nvidia-riva-client>=2.25.1
-Requires-Dist: tokenizers>=0.21.1
+Requires-Dist: tokenizers<0.23,>=0.21.1
 Requires-Dist: huggingface-hub>=0.34.0
 Provides-Extra: service
 Requires-Dist: backoff; extra == "service"
@@ -288,17 +288,21 @@ chunks = ingestor.ingest()  # pandas.DataFrame (batch and inprocess)
 
 ### Ingest a test corpus (CLI)
 
-Point `retriever ingest` at a **directory** of PDFs to produce a ready-to-query
-LanceDB table.
+`retriever ingest` accepts a file or a directory and writes a ready-to-query
+LanceDB table. From a clone of this repository, `./data/multimodal_test.pdf`
+is a valid first-run input. Replace `/path/to/file-or-directory` below with
+that PDF or with your own file or directory.
 
-> **Corpus size matters.** LanceDB's default IVF index needs at least 16
-> chunks to train its 16 k-means partitions. Single-PDF ingestion will fail
-> at the indexing step; point `retriever ingest` at a directory with enough
-> documents to clear that threshold. Replace `/your-example-dir` below with
-> the path to your own corpus.
+> **Small corpora are supported.** The NeMo Retriever LanceDB adapter requests
+> 16 IVF partitions by default. When the table has fewer rows, the adapter
+> clamps that count to one less than the row count. An ingest that produces
+> 2 through 15 rows still builds an IVF index. A one-row table is stored
+> without a vector index, and ingest succeeds. An ingest that produces zero rows
+> is a separate empty-result error. Use a larger corpus when you want more
+> representative retrieval quality.
 
 ```bash
-retriever ingest /your-example-dir \
+retriever ingest /path/to/file-or-directory \
   --lancedb-uri lancedb \
   --table-name nemo-retriever
 ```
@@ -306,8 +310,7 @@ retriever ingest /your-example-dir \
 Chunks land at `./lancedb/nemo-retriever`, which matches the storage settings
 used in [Run a recall query](#run-a-recall-query) below. With the
 `[local]` extra installed (see setup), defaults point at local-GPU extraction
-and embedding. Use enough documents in the directory to clear the LanceDB IVF
-training threshold described above.
+and embedding.
 
 **No local GPU?** Set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)) and route extraction and embedding
 through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:
@@ -315,7 +318,7 @@ through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:
 ```bash
 export NVIDIA_API_KEY=nvapi-...
 
-retriever ingest /your-example-dir \
+retriever ingest /path/to/file-or-directory \
   --lancedb-uri lancedb \
   --table-name nemo-retriever \
   --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
@@ -467,6 +470,8 @@ embedding endpoint.
 **Local in-process vLLM agent LLM.** Omit `--agentic-invoke-url` to load the
 supported local agent LLM directly in the Python process. `nemotron-8b` is the
 default; `super-49b` is also supported when the process has enough visible GPUs.
+For `super-49b`, set `--agentic-local-tensor-parallel-size 2` with two visible
+GPUs (for example `CUDA_VISIBLE_DEVICES=0,1`).
 
 ```bash
 CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
@@ -476,6 +481,15 @@ CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is
   --embed-model-name nvidia/llama-nemotron-embed-1b-v2
 ```
 
+```bash
+CUDA_VISIBLE_DEVICES=0,1 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
+  --agentic \
+  --agentic-llm-model super-49b \
+  --agentic-local-tensor-parallel-size 2 \
+  --lancedb-uri lancedb \
+  --table-name nemo-retriever \
+  --embed-model-name nvidia/llama-nemotron-embed-vl-1b-v2
+```
 **OpenAI-compatible agent endpoint.** Pass `--agentic-invoke-url` when you want a
 custom model or a separately hosted chat-completions server, such as vLLM server
 mode or a self-hosted NIM. When an invoke URL is provided, `--agentic-llm-model`

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/README.md

@@ -178,17 +178,21 @@ chunks = ingestor.ingest()  # pandas.DataFrame (batch and inprocess)
 
 ### Ingest a test corpus (CLI)
 
-Point `retriever ingest` at a **directory** of PDFs to produce a ready-to-query
-LanceDB table.
-
-> **Corpus size matters.** LanceDB's default IVF index needs at least 16
-> chunks to train its 16 k-means partitions. Single-PDF ingestion will fail
-> at the indexing step; point `retriever ingest` at a directory with enough
-> documents to clear that threshold. Replace `/your-example-dir` below with
-> the path to your own corpus.
+`retriever ingest` accepts a file or a directory and writes a ready-to-query
+LanceDB table. From a clone of this repository, `./data/multimodal_test.pdf`
+is a valid first-run input. Replace `/path/to/file-or-directory` below with
+that PDF or with your own file or directory.
+
+> **Small corpora are supported.** The NeMo Retriever LanceDB adapter requests
+> 16 IVF partitions by default. When the table has fewer rows, the adapter
+> clamps that count to one less than the row count. An ingest that produces
+> 2 through 15 rows still builds an IVF index. A one-row table is stored
+> without a vector index, and ingest succeeds. An ingest that produces zero rows
+> is a separate empty-result error. Use a larger corpus when you want more
+> representative retrieval quality.
 
 ```bash
-retriever ingest /your-example-dir \
+retriever ingest /path/to/file-or-directory \
   --lancedb-uri lancedb \
   --table-name nemo-retriever
 ```
@@ -196,8 +200,7 @@ retriever ingest /your-example-dir \
 Chunks land at `./lancedb/nemo-retriever`, which matches the storage settings
 used in [Run a recall query](#run-a-recall-query) below. With the
 `[local]` extra installed (see setup), defaults point at local-GPU extraction
-and embedding. Use enough documents in the directory to clear the LanceDB IVF
-training threshold described above.
+and embedding.
 
 **No local GPU?** Set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)) and route extraction and embedding
 through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:
@@ -205,7 +208,7 @@ through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:
 ```bash
 export NVIDIA_API_KEY=nvapi-...
 
-retriever ingest /your-example-dir \
+retriever ingest /path/to/file-or-directory \
   --lancedb-uri lancedb \
   --table-name nemo-retriever \
   --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
@@ -357,6 +360,8 @@ embedding endpoint.
 **Local in-process vLLM agent LLM.** Omit `--agentic-invoke-url` to load the
 supported local agent LLM directly in the Python process. `nemotron-8b` is the
 default; `super-49b` is also supported when the process has enough visible GPUs.
+For `super-49b`, set `--agentic-local-tensor-parallel-size 2` with two visible
+GPUs (for example `CUDA_VISIBLE_DEVICES=0,1`).
 
 ```bash
 CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
@@ -366,6 +371,15 @@ CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is
   --embed-model-name nvidia/llama-nemotron-embed-1b-v2
 ```
 
+```bash
+CUDA_VISIBLE_DEVICES=0,1 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
+  --agentic \
+  --agentic-llm-model super-49b \
+  --agentic-local-tensor-parallel-size 2 \
+  --lancedb-uri lancedb \
+  --table-name nemo-retriever \
+  --embed-model-name nvidia/llama-nemotron-embed-vl-1b-v2
+```
 **OpenAI-compatible agent endpoint.** Pass `--agentic-invoke-url` when you want a
 custom model or a separately hosted chat-completions server, such as vLLM server
 mode or a self-hosted NIM. When an invoke URL is provided, `--agentic-llm-model`

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/pyproject.toml

@@ -100,7 +100,9 @@ dependencies = [
   # targets the public NVCF Parakeet endpoint (the default) or any remote NIM.
   "nvidia-riva-client>=2.25.1",
   # Exact tokenizer-only TXT/HTML chunking (no transformers, torch, or weights).
-  "tokenizers>=0.21.1",
+  # Nemotron Parse 2.0's published tokenizer contains NFKC-equivalent added
+  # tokens that tokenizers 0.23 rejects with DuplicatePatternError.
+  "tokenizers>=0.21.1,<0.23",
   "huggingface-hub>=0.34.0",
 ]
 

nemo_retriever-2026.8.14.dev183/src/nemo_retriever/_build_info.py

@@ -0,0 +1,4 @@
+"""Build metadata written by CI before packaging."""
+
+BUILD_GIT_SHA = "95309b7c1f5b4b61109bd538f94a9b064af8476c"
+BUILD_DATE = "2026-08-14T23:39:45Z"

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/cli/query/app.py

@@ -182,7 +182,7 @@ def _local_command(
     reranker_api_key_env: opts.RerankerApiKeyEnvOption = None,
     reranker_model_name: opts.RerankerModelNameOption = None,
     reranker_backend: opts.RerankerBackendOption = None,
-    rerank: opts.RerankOption = False,
+    rerank: opts.RerankOption = None,
     retrieval_mode: opts.RetrievalModeOption = "auto",
     output_format: opts.OutputFormatOption = "hits",
     max_text_chars: opts.MaxTextCharsOption = None,
@@ -193,12 +193,14 @@ def _local_command(
     agentic_react_max_steps: opts.AgenticReactMaxStepsOption = 50,
     agentic_text_truncation: opts.AgenticTextTruncationOption = 0,
     agentic_temperature: opts.AgenticTemperatureOption = None,
+    agentic_local_tensor_parallel_size: opts.AgenticLocalTensorParallelSizeOption = 1,
     agentic_llm_client: opts.AgenticLlmClientOption = None,
 ) -> None:
     _validate_output_options(output_format, max_text_chars)
     if reranker_invoke_url is None:
         reranker_invoke_url = os.environ.get("RERANKER_INVOKE_URL") or None
-    rerank = rerank or bool(reranker_invoke_url) or bool(reranker_model_name) or bool(reranker_backend)
+    if rerank is None:
+        rerank = bool(reranker_invoke_url) or bool(reranker_model_name) or bool(reranker_backend)
     silence_noisy_libraries()
     if agentic:
         # Relaxed model gating: an explicit model is required only for the remote
@@ -276,6 +278,7 @@ def _local_command(
                     react_max_steps=agentic_react_max_steps,
                     text_truncation=agentic_text_truncation,
                     temperature=agentic_temperature,
+                    local_tensor_parallel_size=agentic_local_tensor_parallel_size,
                     llm_client=agentic_llm_client,
                 ),
             )

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/cli/query/options.py

@@ -122,12 +122,12 @@ RerankerBackendOption = Annotated[
     ),
 ]
 RerankOption = Annotated[
-    bool,
+    bool | None,
     typer.Option(
         "--rerank/--no-rerank",
         help=(
-            "Enable reranking after vector retrieval. Default off. Implicitly enabled when "
-            "any of --reranker-invoke-url / --reranker-model-name / --reranker-backend is set."
+            "Enable reranking after vector retrieval. Default off. When neither flag is passed, implicitly enabled "
+            "when any of --reranker-invoke-url / --reranker-model-name / --reranker-backend is set."
         ),
     ),
 ]
@@ -216,6 +216,18 @@ AgenticTemperatureOption = Annotated[
         ),
     ),
 ]
+AgenticLocalTensorParallelSizeOption = Annotated[
+    int,
+    typer.Option(
+        "--agentic-local-tensor-parallel-size",
+        min=1,
+        help=(
+            "vLLM tensor_parallel_size for the in-process agent LLM. "
+            "Use 2 (with two visible GPUs) for local super-49b; ignored when "
+            "--agentic-invoke-url is set."
+        ),
+    ),
+]
 AgenticLlmClientOption = Annotated[
     str | None,
     typer.Option(

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/common/api/internal/schemas/meta/metadata_schema.py

@@ -189,6 +189,9 @@ class ImageMetadataSchema(BaseModelNoExt):
     uploaded_image_url: str = ""
     """A mirror of source_metadata.source_location."""
 
+    uploaded_image_uri: str = ""
+    """The URI of the stored image asset, when image storage is configured."""
+
     width: int = 0
     """The width of the image."""
 

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/common/modality/txt/split.py

@@ -166,7 +166,6 @@ def split_df(
                 meta["chunk_index"] = i
                 meta["chunk_count"] = len(chunks)
                 meta["content"] = chunk
-            new_row["page_number"] = i + 1
             # Only the first chunk keeps structured content so that
             # downstream explode does not duplicate tables/charts/etc.
             if i > 0:

nemo_retriever-2026.8.14.dev183/src/nemo_retriever/common/stage_errors.py

@@ -0,0 +1,46 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
+# All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared detection of structured row-level stage errors."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import Any
+
+ERROR_FIELD_KEYS = ("error", "errors", "exception", "traceback", "failed")
+
+
+def is_populated_error_field(key: str, value: Any) -> bool:
+    """Return whether a conventional stage-error field contains a failure."""
+    if value is None:
+        return False
+    if key == "failed" and isinstance(value, bool):
+        return value
+    if isinstance(value, str):
+        return bool(value.strip())
+    if isinstance(value, (list, tuple, set, dict)):
+        return len(value) > 0
+    return bool(value)
+
+
+def iter_stage_errors_from_value(value: Any, *, path: str = "") -> Iterator[dict[str, Any]]:
+    """Yield populated error fields recursively with their value paths."""
+    if isinstance(value, dict):
+        for key in ERROR_FIELD_KEYS:
+            if key in value and is_populated_error_field(key, value.get(key)):
+                yield {
+                    "path": f"{path}.{key}" if path else key,
+                    "error": value.get(key),
+                }
+        for key, child in value.items():
+            if key in ERROR_FIELD_KEYS and is_populated_error_field(key, child):
+                continue
+            child_path = f"{path}.{key}" if path else str(key)
+            yield from iter_stage_errors_from_value(child, path=child_path)
+        return
+    if isinstance(value, (list, tuple)):
+        for index, child in enumerate(value):
+            child_path = f"{path}[{index}]" if path else f"[{index}]"
+            yield from iter_stage_errors_from_value(child, path=child_path)

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/common/vdb/records.py

@@ -7,6 +7,7 @@
 from __future__ import annotations
 
 import json
+from collections import Counter
 from collections.abc import Mapping
 from pathlib import Path
 from typing import Any, TypedDict
@@ -14,6 +15,7 @@ from typing import Any, TypedDict
 from pydantic import ValidationError
 
 from nemo_retriever.common.schemas.collections import QueryHit
+from nemo_retriever.common.stage_errors import ERROR_FIELD_KEYS, iter_stage_errors_from_value
 
 _CONTENT_TYPE_ALIASES: dict[str, str] = {
     "chart_caption": "chart",
@@ -34,6 +36,10 @@ class RetrievalContractError(RuntimeError):
     """A backend retrieval result cannot satisfy the canonical hit contract."""
 
 
+class VdbUploadError(ValueError):
+    """A nonempty graph batch cannot produce any canonical VDB records."""
+
+
 def validate_collection_retrieval_results(
     result: Any,
     *,
@@ -240,6 +246,16 @@ def _is_image_backed_row(row: dict[str, Any]) -> bool:
     )
 
 
+def _is_inherited_page_uri(row: dict[str, Any], stored_image_uri: str, content_type: str | None) -> bool:
+    """Return whether a structured row only carries its page image URI."""
+    if content_type not in {"table", "chart", "infographic"}:
+        return False
+
+    page_image = row.get("page_image")
+    page_uri = page_image.get("stored_image_uri") if isinstance(page_image, dict) else None
+    return bool(_first_str(page_uri) == stored_image_uri)
+
+
 def _derive_fidelity(content_type: Any, metadata: dict[str, Any], content_metadata: dict[str, Any]) -> str | None:
     """Map a chunk's modality + real provenance signals to a trust tier.
 
@@ -290,6 +306,8 @@ def _client_record_from_graph_row(row: dict[str, Any], *, require_embedding: boo
     stored_image_uri = _first_str(row.get("_stored_image_uri"), row.get("stored_image_uri"))
     if stored_image_uri:
         content_metadata.setdefault("stored_image_uri", stored_image_uri)
+        if not _is_inherited_page_uri(row, stored_image_uri, content_type):
+            content_metadata.setdefault("uploaded_image_uri", stored_image_uri)
     bbox = _bbox_from_graph_row(row)
     if bbox is not None:
         content_metadata.setdefault("bbox_xyxy_norm", bbox)
@@ -336,19 +354,61 @@ def _row_has_uploadable_content_without_embedding(row: dict[str, Any]) -> bool:
     return bool(_text_from_graph_row(row, metadata)) or _is_image_backed_row(row)
 
 
+def _stage_error_field(path: Any) -> str:
+    text = str(path or "")
+    for field in ERROR_FIELD_KEYS:
+        if text == field or text.endswith(f".{field}"):
+            return field
+    return "error"
+
+
+def _raise_for_empty_vdb_conversion(graph_rows: list[dict[str, Any]]) -> None:
+    upstream_errors = [error for row in graph_rows for error in iter_stage_errors_from_value(row)]
+    if upstream_errors:
+        error_fields = Counter(_stage_error_field(error.get("path")) for error in upstream_errors)
+        summary = ", ".join(f"{field}={count}" for field, count in sorted(error_fields.items()))
+        raise VdbUploadError(
+            f"vdb_upload received {len(graph_rows)} row(s), but none were uploadable because upstream stages "
+            f"reported {len(upstream_errors)} structured row error(s) ({summary}); "
+            "error payloads are omitted because they may contain sensitive data."
+        )
+
+    reasons = Counter(
+        (
+            "missing embedding"
+            if _row_has_uploadable_content_without_embedding(row)
+            else "missing searchable text or image backing"
+        )
+        for row in graph_rows
+    )
+    if "missing embedding" in reasons:
+        summary = ", ".join(f"{reason}={count}" for reason, count in sorted(reasons.items()))
+        raise VdbUploadError(
+            "vdb_upload requires embedded records, but no embeddings were found. "
+            f"Received {len(graph_rows)} nonempty row(s); rejection reasons: {summary}. "
+            "Add an embed stage or provide a supported embedding column."
+        )
+    summary = ", ".join(f"{reason}={count}" for reason, count in sorted(reasons.items()))
+    raise VdbUploadError(
+        f"vdb_upload received {len(graph_rows)} row(s), but none were uploadable; rejection reasons: {summary}."
+    )
+
+
 def to_client_vdb_records(rows: Any) -> list[list[dict[str, Any]]]:
     """Convert graph-ingest rows into the nested record shape expected by client VDBs.
 
     Dense rows require an embedding and either nonblank text or concrete image backing.
-    When no row survives conversion, returns ``[]`` — a falsy value so
-    ``if not records`` skips :meth:`~nemo_retriever.vdb.adt_vdb.VDB.run`.
+    Genuinely empty input returns ``[]`` so Ray can process empty partitions.
+    A nonempty graph batch that produces zero records raises
+    :class:`VdbUploadError` before a backend write is attempted.
     When at least one row converts, returns ``[batch]`` with a single non-empty inner list
     (never ``[[]]``, which would be truthy and could trip backends on an empty insert).
-    Uploadable graph content without an embedding raises ``ValueError`` when no
-    row survives conversion. Genuinely empty input continues to return ``[]``.
+    Uploadable graph content without an embedding raises ``VdbUploadError`` when
+    no row survives conversion.
     """
     if isinstance(rows, list) and all(isinstance(batch, list) for batch in rows):
-        return rows
+        nonempty_batches = [batch for batch in rows if batch]
+        return rows if len(nonempty_batches) == len(rows) else nonempty_batches
     if hasattr(rows, "to_pandas"):
         rows = rows.to_pandas()
     if hasattr(rows, "to_dict"):
@@ -358,11 +418,8 @@ def to_client_vdb_records(rows: Any) -> list[list[dict[str, Any]]]:
     # would call _client_record_from_graph_row twice per row on large datasets.
     # isinstance(row, dict): plain lists are not normalized like DataFrame rows; skip None/Series/etc.
     inner = [record for row in graph_rows if (record := _client_record_from_graph_row(row)) is not None]
-    if not inner and any(_row_has_uploadable_content_without_embedding(row) for row in graph_rows):
-        raise ValueError(
-            "vdb_upload requires embedded records, but no embeddings were found. "
-            "Add an embed stage or provide a supported embedding column."
-        )
+    if not inner and graph_rows:
+        _raise_for_empty_vdb_conversion(graph_rows)
     # Preserve legacy contract: no uploadable rows → [], not [[]].
     return [inner] if inner else []
 

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/ingest/plan.py

@@ -604,6 +604,15 @@ def resolve_ingest_plan(request: IngestPlanRequest) -> ResolvedIngestPlan:
     _validate_profile_manifest(validated_profile, branches)
 
     extract_kwargs = profile_extract_defaults(validated_profile)
+    if (
+        validated_profile == "auto"
+        and extract.method is None
+        and (extract.ocr_version is not None or extract.ocr_lang is not None)
+    ):
+        # An explicit OCR selector is an opt-in to scanned-page OCR. The
+        # default pdfium method only reads a native text layer, so select the
+        # hybrid method unless the caller chose a method explicitly.
+        extract_kwargs["method"] = "pdfium_hybrid"
     extract_kwargs.update(
         {
             key: value

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/ingestor/core.py

@@ -102,6 +102,13 @@ class ingestor:
             f"{self.__class__.__name__}.{method_name}() is not implemented yet " f"(run_mode={self.RUN_MODE})."
         )
 
+    def _validate_input_sources(self, inline_texts: Sequence[str] | None) -> None:
+        if self._documents or self._buffers or inline_texts:
+            return
+        raise ValueError(
+            "No input sources configured. Call files(), texts(), or buffers() with at least one source before ingest()."
+        )
+
     def files(self, documents: Union[str, List[str]]) -> "ingestor":
         """Add document paths/URIs for processing."""
         self._not_implemented("files")

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/ingestor/graph_ingestor.py

@@ -84,13 +84,19 @@ from nemo_retriever.common.input_files import (
 from nemo_retriever.common.remote_auth import resolve_remote_api_key
 from nemo_retriever.common.ray_runtime import ensure_local_ray_runtime
 from nemo_retriever.common.ray_resource_hueristics import gather_cluster_resources
+from nemo_retriever.common.stage_errors import (
+    ERROR_FIELD_KEYS,
+    is_populated_error_field,
+    iter_stage_errors_from_value,
+)
 from nemo_retriever.common.modality.txt.split import empty_text_chunks_df
 
-_ERROR_FIELD_KEYS = ("error", "errors", "exception", "traceback", "failed")
+_ERROR_FIELD_KEYS = ERROR_FIELD_KEYS
 _REMOTE_EMBED_ENDPOINT_FIELDS = ("embedding_endpoint", "embed_invoke_url")
 _DEFAULT_PAGE_ELEMENTS_COLUMN = "page_elements_v3"
 _DEFAULT_EMBED_COLUMN = "text_embeddings_1b_v2"
 _ERROR_MESSAGE_LIMIT = 256
+_SOURCE_IDENTIFIER_COLUMNS = ("document_id", "path", "source_path", "metadata", "source_id", "source_name")
 logger = logging.getLogger(__name__)
 _HTTP_STATUS_FIELDS: tuple[str, ...] = ("status_code", "http_status", "status", "code")
 _EXPLICIT_MODE_INPUT_TYPES: dict[str, frozenset[str]] = {
@@ -777,6 +783,7 @@ class GraphIngestor(ingestor):
             service-style ``(source, error)`` tuples.
         """
         return_failures = self._resolve_return_failures(params, kwargs)
+        self._validate_input_sources(self._inline_texts)
         if not self._documents and not self._buffers and is_blank_inline_corpus(self._inline_texts):
             result = empty_text_chunks_df()
             if self._run_mode == "batch":
@@ -1104,35 +1111,11 @@ class GraphIngestor(ingestor):
 
     @staticmethod
     def _is_populated_error_field(key: str, value: Any) -> bool:
-        if value is None:
-            return False
-        if key == "failed" and isinstance(value, bool):
-            return value
-        if isinstance(value, str):
-            return bool(value.strip())
-        if isinstance(value, (list, tuple, set, dict)):
-            return len(value) > 0
-        return bool(value)
+        return is_populated_error_field(key, value)
 
     @classmethod
     def _iter_stage_errors_from_value(cls, value: Any, *, path: str = "") -> Iterator[dict[str, Any]]:
-        if isinstance(value, dict):
-            for key in _ERROR_FIELD_KEYS:
-                if key in value and cls._is_populated_error_field(key, value.get(key)):
-                    yield {
-                        "path": f"{path}.{key}" if path else key,
-                        "error": value.get(key),
-                    }
-            for key, child in value.items():
-                if key in _ERROR_FIELD_KEYS and cls._is_populated_error_field(key, child):
-                    continue
-                child_path = f"{path}.{key}" if path else str(key)
-                yield from cls._iter_stage_errors_from_value(child, path=child_path)
-            return
-        if isinstance(value, (list, tuple)):
-            for i, child in enumerate(value):
-                child_path = f"{path}[{i}]" if path else f"[{i}]"
-                yield from cls._iter_stage_errors_from_value(child, path=child_path)
+        yield from iter_stage_errors_from_value(value, path=path)
 
     @staticmethod
     def _row_value(row: Any, key: str) -> Any:
@@ -1201,14 +1184,15 @@ class GraphIngestor(ingestor):
             return []
         requested_columns = list(columns) if columns is not None else None
 
-        if callable(iter_batches):
-            batches = (arrow_table_to_pandas(batch_df) for batch_df in iter_batches(batch_format="pyarrow"))
-        else:
-            batches = (batch,)
+        batches = iter_batches(batch_format="pyarrow") if callable(iter_batches) else (batch,)
 
         records: list[dict[str, Any]] = []
-        for batch_df in batches:
-            available_columns = getattr(batch_df, "columns", None)
+        for raw_batch in batches:
+            available_columns = (
+                getattr(raw_batch, "column_names", None)
+                if callable(iter_batches)
+                else getattr(raw_batch, "columns", None)
+            )
             if available_columns is None:
                 continue
             target_columns = (
@@ -1216,11 +1200,24 @@ class GraphIngestor(ingestor):
                 if requested_columns is None
                 else [c for c in requested_columns if c in available_columns]
             )
-            # ``iterrows`` materializes the full frame through NumPy, which is
-            # unsafe for Ray pickled-object columns containing empty arrays.
-            row_values = batch_df.itertuples(index=False, name=None)
-            for row_index, values in zip(batch_df.index, row_values):
-                row = dict(zip(available_columns, values))
+            if not target_columns:
+                continue
+            # Finalization only needs diagnostic payloads and source context.
+            # Reading the full frame can iterate unrelated Arrow-backed image
+            # columns that pandas cannot safely materialize row by row.
+            scan_columns = list(
+                dict.fromkeys(
+                    [*target_columns, *(column for column in _SOURCE_IDENTIFIER_COLUMNS if column in available_columns)]
+                )
+            )
+            batch_df = (
+                arrow_table_to_pandas(raw_batch.select(scan_columns))
+                if callable(iter_batches)
+                else raw_batch.loc[:, scan_columns]
+            )
+            column_values = {column: batch_df[column].array for column in scan_columns}
+            for row_position, row_index in enumerate(batch_df.index):
+                row = {column: values[row_position] for column, values in column_values.items()}
                 source_identifier = cls._source_identifier_from_row(row, row_index)
                 for column in target_columns:
                     for record in cls._iter_stage_errors_from_value(row[column]):

{nemo_retriever-2026.8.13.dev182 → nemo_retriever-2026.8.14.dev183}/src/nemo_retriever/models/hf_model_registry.py

@@ -25,14 +25,14 @@ logger = logging.getLogger(__name__)
 
 HF_MODEL_REVISIONS: dict[str, str] = {
     "nvidia/llama-3.2-nv-embedqa-1b-v2": "cefc2394cc541737b7867df197984cf23f05367f",
-    "nvidia/llama-nemotron-embed-1b-v2": "b4caa8456edd360b3b4e938d94ed4398dd437fad",
+    "nvidia/llama-nemotron-embed-1b-v2": "113abe4acafa848e77ead9c0623205e511932348",
     "nvidia/parakeet-ctc-1.1b": "a707e818195cb97c8f7da2fc36b221a29f69a5db",
     "nvidia/NVIDIA-Nemotron-Parse-v1.2": "f42c8040b12ee64370922d108778ab655b722c5d",
     "nvidia/llama-nemotron-embed-vl-1b-v2": "171968f5db67b2d16aac89b820d53b4eb6a7ab44",
     "meta-llama/Llama-3.2-1B": "4e20de362430cd3b72f300e6b0f18e50e7166e08",
     "intfloat/e5-large-unsupervised": "15af9288f69a6291f37bfb89b47e71abc747b206",
     "nvidia/llama-nemotron-rerank-1b-v2": "8fd3e5d962d44cfe65d4ba0784eebed44cf136b0",
-    "nvidia/llama-nemotron-rerank-vl-1b-v2": "edc083f4b3a433d65287cbca916759c9f88fa887",
+    "nvidia/llama-nemotron-rerank-vl-1b-v2": "9c20c4aedf9ec87b6b7346c3bc4754ea030dab35",
     "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16": "5d250e2e111dc5e1434131bdf3d590c27a878ade",
     "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8": "7394488badb786e1decc0e00e308de1cab9560e6",
     "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD": "b8d3c170d9ee3a078917ef9bfd508eff988d6de7",