nemo-retriever 2026.6.28.dev136__tar.gz → 2026.6.29.dev137__tar.gz

{nemo_retriever-2026.6.28.dev136/src/nemo_retriever.egg-info → nemo_retriever-2026.6.29.dev137}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nemo-retriever
-Version: 2026.6.28.dev136
+Version: 2026.6.29.dev137
 Summary: A modern RAG ingestion pipeline from Nvidia
 Author-email: Jeremy Dyer <jdyer@nvidia.com>
 License-Expression: Apache-2.0
@@ -420,83 +420,6 @@ Answer:
 Cat is the animal whose activity (jumping onto a laptop) matches the location of the typos, so the cat is responsible for the typos in the documents.
 ```
 
-### Run agentic retrieval
-
-Agentic retrieval runs an LLM-driven ReAct loop over an existing LanceDB index.
-It does not ingest documents; first build the index with one of the ingestion
-flows above, then query the same `lancedb_uri`, `table_name`, and embedding
-model.
-
-For [build.nvidia.com](https://build.nvidia.com/) hosted inference, set
-`NVIDIA_API_KEY`. On CPU-only machines, the CPU embedding actor and agent LLM
-use the hosted NVIDIA endpoints by default:
-
-```bash
-export NVIDIA_API_KEY=nvapi-...
-
-retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
-  --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
-  --lancedb-uri lancedb \
-  --table-name nemo-retriever \
-  --embed-model-name nvidia/llama-nemotron-embed-1b-v2
-```
-
-The agentic LLM uses the built-in NVIDIA hosted chat-completions endpoint when
-`--agentic-invoke-url` is omitted. On CPU-only machines, embedding actors also
-resolve to CPU/remote implementations and default to hosted endpoints. On
-GPU-capable machines, embedding prefers the local GPU implementation unless an
-endpoint URL, such as `--embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings`,
-is provided.
-
-For a quick smoke test, lower the amount of agent work:
-
-```bash
-retriever query "What is RAG?" \
-  --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
-  --lancedb-uri lancedb \
-  --table-name nemo-retriever \
-  --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
-  --top-k 1 \
-  --agentic-react-max-steps 1 \
-  --agentic-backend-top-k 1
-```
-
-The same flow is available from Python. It uses the same `NVIDIA_API_KEY`
-environment variable shown above for hosted embedding and chat-completions
-requests.
-
-```python
-from nemo_retriever.cli.query_workflow import agentic_query_documents
-from nemo_retriever.query.options import (
-    QueryAgenticOptions,
-    QueryEmbedOptions,
-    QueryRequest,
-    QueryRetrievalOptions,
-    QueryStorageOptions,
-)
-
-# Requires NVIDIA_API_KEY=nvapi-... in the environment.
-results = agentic_query_documents(
-    QueryRequest(
-        query="What is RAG?",
-        retrieval=QueryRetrievalOptions(top_k=10),
-        storage=QueryStorageOptions(
-            lancedb_uri="lancedb",
-            table_name="nemo-retriever",
-        ),
-        embed=QueryEmbedOptions(
-            embed_model_name="nvidia/llama-nemotron-embed-1b-v2",
-        ),
-        agentic=QueryAgenticOptions(
-            enabled=True,
-            llm_model="nvidia/llama-3.3-nemotron-super-49b-v1.5",
-        ),
-    )
-)
-```
-
 ### Live RAG SDK (retrieve + answer in one call)
 
 The pattern above -- retrieve hits, build a prompt, call an LLM -- is baked into the SDK as `Retriever.answer()` so live applications can skip the boilerplate. The same `Retriever` instance powers three entry points:
@@ -822,26 +745,11 @@ sudo apt install python3.12-dev
 
 After installing the headers, restart the pipeline.
 
-## ViDoRe Harness Sweep
-
-The harness includes BEIR-style ViDoRe dataset presets in `nemo_retriever/harness/test_configs.yaml` and a ready-made sweep definition in `nemo_retriever/harness/vidore_sweep.yaml`.
-
-The ViDoRe harness datasets are configured to:
-
-- read PDFs from `/datasets/retrieval-eval/vidore_v3_corpus_pdf/...`
-- ingest with `embed_modality: text_image`
-- embed at `embed_granularity: page`
-- enable `extract_page_as_image: true` and `extract_infographics: true`
-- evaluate with BEIR-style `ndcg` and `recall` metrics
-
-To run the full ViDoRe sweep:
-
-```bash
-cd ~/NeMo-Retriever/nemo_retriever
-retriever-harness sweep --runs-config harness/vidore_sweep.yaml
-```
+## Retriever Harness
 
-The same commands also work under the main CLI as `retriever harness ...` if you prefer a single top-level command namespace.
+The developer harness runs code-owned benchmarks through `retriever harness`.
+Use `retriever harness list --runsets` to see available benchmark names and
+runsets, then run one benchmark with `retriever harness run <benchmark>`.
 
 ### Ingest image storage
 
@@ -855,6 +763,4 @@ retriever ingest ./data \
 
 The store stage writes the image payloads produced by ingest. With
 `--embed-granularity page`, stored assets are page images. With
-`--embed-granularity element`, stored assets are element images. Store is not
-currently configured through the harness; use `retriever pipeline run` only when
-you also need pipeline-specific compatibility artifacts.
+`--embed-granularity element`, stored assets are element images.

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/README.md

@@ -309,83 +309,6 @@ Answer:
 Cat is the animal whose activity (jumping onto a laptop) matches the location of the typos, so the cat is responsible for the typos in the documents.
 ```
 
-### Run agentic retrieval
-
-Agentic retrieval runs an LLM-driven ReAct loop over an existing LanceDB index.
-It does not ingest documents; first build the index with one of the ingestion
-flows above, then query the same `lancedb_uri`, `table_name`, and embedding
-model.
-
-For [build.nvidia.com](https://build.nvidia.com/) hosted inference, set
-`NVIDIA_API_KEY`. On CPU-only machines, the CPU embedding actor and agent LLM
-use the hosted NVIDIA endpoints by default:
-
-```bash
-export NVIDIA_API_KEY=nvapi-...
-
-retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
-  --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
-  --lancedb-uri lancedb \
-  --table-name nemo-retriever \
-  --embed-model-name nvidia/llama-nemotron-embed-1b-v2
-```
-
-The agentic LLM uses the built-in NVIDIA hosted chat-completions endpoint when
-`--agentic-invoke-url` is omitted. On CPU-only machines, embedding actors also
-resolve to CPU/remote implementations and default to hosted endpoints. On
-GPU-capable machines, embedding prefers the local GPU implementation unless an
-endpoint URL, such as `--embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings`,
-is provided.
-
-For a quick smoke test, lower the amount of agent work:
-
-```bash
-retriever query "What is RAG?" \
-  --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
-  --lancedb-uri lancedb \
-  --table-name nemo-retriever \
-  --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
-  --top-k 1 \
-  --agentic-react-max-steps 1 \
-  --agentic-backend-top-k 1
-```
-
-The same flow is available from Python. It uses the same `NVIDIA_API_KEY`
-environment variable shown above for hosted embedding and chat-completions
-requests.
-
-```python
-from nemo_retriever.cli.query_workflow import agentic_query_documents
-from nemo_retriever.query.options import (
-    QueryAgenticOptions,
-    QueryEmbedOptions,
-    QueryRequest,
-    QueryRetrievalOptions,
-    QueryStorageOptions,
-)
-
-# Requires NVIDIA_API_KEY=nvapi-... in the environment.
-results = agentic_query_documents(
-    QueryRequest(
-        query="What is RAG?",
-        retrieval=QueryRetrievalOptions(top_k=10),
-        storage=QueryStorageOptions(
-            lancedb_uri="lancedb",
-            table_name="nemo-retriever",
-        ),
-        embed=QueryEmbedOptions(
-            embed_model_name="nvidia/llama-nemotron-embed-1b-v2",
-        ),
-        agentic=QueryAgenticOptions(
-            enabled=True,
-            llm_model="nvidia/llama-3.3-nemotron-super-49b-v1.5",
-        ),
-    )
-)
-```
-
 ### Live RAG SDK (retrieve + answer in one call)
 
 The pattern above -- retrieve hits, build a prompt, call an LLM -- is baked into the SDK as `Retriever.answer()` so live applications can skip the boilerplate. The same `Retriever` instance powers three entry points:
@@ -711,26 +634,11 @@ sudo apt install python3.12-dev
 
 After installing the headers, restart the pipeline.
 
-## ViDoRe Harness Sweep
-
-The harness includes BEIR-style ViDoRe dataset presets in `nemo_retriever/harness/test_configs.yaml` and a ready-made sweep definition in `nemo_retriever/harness/vidore_sweep.yaml`.
-
-The ViDoRe harness datasets are configured to:
-
-- read PDFs from `/datasets/retrieval-eval/vidore_v3_corpus_pdf/...`
-- ingest with `embed_modality: text_image`
-- embed at `embed_granularity: page`
-- enable `extract_page_as_image: true` and `extract_infographics: true`
-- evaluate with BEIR-style `ndcg` and `recall` metrics
-
-To run the full ViDoRe sweep:
-
-```bash
-cd ~/NeMo-Retriever/nemo_retriever
-retriever-harness sweep --runs-config harness/vidore_sweep.yaml
-```
+## Retriever Harness
 
-The same commands also work under the main CLI as `retriever harness ...` if you prefer a single top-level command namespace.
+The developer harness runs code-owned benchmarks through `retriever harness`.
+Use `retriever harness list --runsets` to see available benchmark names and
+runsets, then run one benchmark with `retriever harness run <benchmark>`.
 
 ### Ingest image storage
 
@@ -744,6 +652,4 @@ retriever ingest ./data \
 
 The store stage writes the image payloads produced by ingest. With
 `--embed-granularity page`, stored assets are page images. With
-`--embed-granularity element`, stored assets are element images. Store is not
-currently configured through the harness; use `retriever pipeline run` only when
-you also need pipeline-specific compatibility artifacts.
+`--embed-granularity element`, stored assets are element images.

nemo_retriever-2026.6.29.dev137/src/nemo_retriever/_build_info.py

@@ -0,0 +1,4 @@
+"""Build metadata written by CI before packaging."""
+
+BUILD_GIT_SHA = "18d3bbd99b81cff0748cc142397587b14d49157b"
+BUILD_DATE = "2026-06-29T23:51:08Z"

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/cli/ingest/app.py

@@ -10,6 +10,7 @@ from typer.core import TyperGroup
 
 from nemo_retriever.cli.ingest.graph_commands import _graph_ingest_command
 from nemo_retriever.cli.ingest.service import _service_command
+from nemo_retriever.cli.ingest.options import DEFAULT_CAPTION_MODEL, DEFAULT_EMBED_MODEL
 
 _DEFAULT_COMMAND = "local"
 _GROUP_OPTIONS = {"--help", "-h"}
@@ -31,6 +32,18 @@ app = typer.Typer(
     no_args_is_help=True,
 )
 
-app.command("local")(_graph_ingest_command)
-app.command("batch")(_graph_ingest_command)
+app.command(
+    "local",
+    help=(
+        f"Run local ingest into a LanceDB index. Default embedding model: {DEFAULT_EMBED_MODEL}. "
+        f"Default caption model when captioning: {DEFAULT_CAPTION_MODEL}."
+    ),
+)(_graph_ingest_command)
+app.command(
+    "batch",
+    help=(
+        f"Run Ray batch ingest into a LanceDB index. Default embedding model: {DEFAULT_EMBED_MODEL}. "
+        f"Default caption model when captioning: {DEFAULT_CAPTION_MODEL}."
+    ),
+)(_graph_ingest_command)
 app.command("service")(_service_command)

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/cli/ingest/options.py

@@ -8,6 +8,7 @@ from typing import Annotated
 
 import typer
 
+from nemo_retriever.common.params import CaptionParams
 from nemo_retriever.ingest.plan import (
     AudioSplitTypeValue,
     IngestIndexModeValue,
@@ -17,6 +18,10 @@ from nemo_retriever.ingest.plan import (
     OcrVersionValue,
     TableOutputFormatValue,
 )
+from nemo_retriever.models import VL_EMBED_MODEL
+
+DEFAULT_EMBED_MODEL = VL_EMBED_MODEL
+DEFAULT_CAPTION_MODEL = CaptionParams().model_name
 
 DocumentsArgument = Annotated[
     list[str],
@@ -163,7 +168,13 @@ ApiKeyOption = Annotated[
 ]
 CaptionModelNameOption = Annotated[
     str | None,
-    typer.Option("--caption-model-name", help="Optional VLM caption model name override."),
+    typer.Option(
+        "--caption-model-name",
+        help=(
+            f"Optional VLM caption model name override. Defaults to {DEFAULT_CAPTION_MODEL} "
+            "when --caption is enabled."
+        ),
+    ),
 ]
 CaptionContextTextMaxCharsOption = Annotated[
     int | None,
@@ -272,7 +283,10 @@ TableOutputFormatOption = Annotated[
 EmbedInvokeUrlOption = Annotated[str | None, typer.Option("--embed-invoke-url", help="Embedding NIM endpoint URL.")]
 EmbedModelNameOption = Annotated[
     str | None,
-    typer.Option("--embed-model-name", help="Optional embedding model name override."),
+    typer.Option(
+        "--embed-model-name",
+        help=f"Optional embedding model name override. Defaults to {DEFAULT_EMBED_MODEL} when omitted.",
+    ),
 ]
 LocalIngestEmbedBackendOption = Annotated[
     LocalIngestEmbedBackendValue | None,

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/cli/pdf/stage.py

@@ -554,7 +554,10 @@ def render_page_elements(
     nemotron_parse_model_name: Optional[str] = typer.Option(
         None,
         "--nemotron-parse-model-name",
-        help="Nemotron Parse model name (optional; defaults to schema default).",
+        help=(
+            "Nemotron Parse model name. Used only with --method nemotron_parse; "
+            "omitted value uses that method's schema default."
+        ),
     ),
     extract_text: bool = typer.Option(True, "--extract-text/--no-extract-text", help="Extract text primitives."),
     extract_images: bool = typer.Option(
@@ -705,13 +708,15 @@ def render_page_elements(
                 "Nemotron Parse endpoint required for method 'nemotron_parse'. "
                 "Set --nemotron-parse-grpc-endpoint or --nemotron-parse-http-endpoint."
             )
-        extractor_cfg["nemotron_parse_config"] = {
+        nemotron_parse_config = {
             "auth_token": auth_token,
             # Nemotron Parse may still rely on YOLOX for region proposals depending on config.
             "yolox_endpoints": [yolox_grpc_endpoint, yolox_http_endpoint],
             "nemotron_parse_endpoints": [nemotron_parse_grpc_endpoint, nemotron_parse_http_endpoint],
-            "nemotron_parse_model_name": nemotron_parse_model_name,
         }
+        if nemotron_parse_model_name is not None:
+            nemotron_parse_config["nemotron_parse_model_name"] = nemotron_parse_model_name
+        extractor_cfg["nemotron_parse_config"] = nemotron_parse_config
 
     extractor_schema = load_pdf_extractor_schema_from_dict(extractor_cfg)
     task_cfg = make_pdf_task_config(

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/cli/query/app.py

@@ -155,7 +155,14 @@ def _retrieval_options(
     )
 
 
-@app.command("_local", hidden=True)
+@app.command(
+    "_local",
+    hidden=True,
+    help=(
+        f"Query a local LanceDB index. Default embedding model: {opts.DEFAULT_EMBED_MODEL}. "
+        f"Default local reranker model when reranking: {opts.DEFAULT_RERANK_MODEL}."
+    ),
+)
 def _local_command(
     ctx: typer.Context,
     query: opts.QueryArgument,

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/cli/query/options.py

@@ -8,6 +8,11 @@ from typing import Annotated
 
 import typer
 
+from nemo_retriever.models import VL_EMBED_MODEL, VL_RERANK_MODEL
+
+DEFAULT_EMBED_MODEL = VL_EMBED_MODEL
+DEFAULT_RERANK_MODEL = VL_RERANK_MODEL
+
 
 QueryArgument = Annotated[str, typer.Argument(..., help="Query text.")]
 TopKOption = Annotated[
@@ -62,7 +67,10 @@ EmbedInvokeUrlOption = Annotated[
 ]
 EmbedModelNameOption = Annotated[
     str | None,
-    typer.Option("--embed-model-name", help="Optional embedding model name override."),
+    typer.Option(
+        "--embed-model-name",
+        help=f"Optional embedding model name override. Defaults to {DEFAULT_EMBED_MODEL} when omitted.",
+    ),
 ]
 RerankerInvokeUrlOption = Annotated[
     str | None,
@@ -82,7 +90,7 @@ RerankerModelNameOption = Annotated[
     str | None,
     typer.Option(
         "--reranker-model-name",
-        help="Optional reranker model name override (used by the local GPU reranker).",
+        help=("Optional reranker model name override. When reranking locally, " f"defaults to {DEFAULT_RERANK_MODEL}."),
     ),
 ]
 RerankerBackendOption = Annotated[

nemo_retriever-2026.6.29.dev137/src/nemo_retriever/common/ray_runtime.py

@@ -0,0 +1,82 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-26, NVIDIA CORPORATION & AFFILIATES.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Helpers for starting local Ray runtimes from Retriever processes."""
+
+from __future__ import annotations
+
+import os
+import sys
+from collections.abc import Iterator
+from contextlib import contextmanager
+from typing import Any
+
+from nemo_retriever.common.remote_auth import collect_remote_auth_runtime_env
+from nemo_retriever.models.hf_cache import collect_hf_runtime_env
+
+
+_UV_ENV_VARS = ("UV", "UV_RUN_RECURSION_DEPTH")
+_RAY_UV_RUNTIME_ENV_FLAG = "RAY_ENABLE_UV_RUN_RUNTIME_ENV"
+
+
+@contextmanager
+def without_uv_run_env() -> Iterator[None]:
+    """Prevent Ray workers from recursively bootstrapping through ``uv run``."""
+
+    previous = {name: os.environ.pop(name) for name in _UV_ENV_VARS if name in os.environ}
+    previous_ray_flag = os.environ.get(_RAY_UV_RUNTIME_ENV_FLAG)
+    os.environ[_RAY_UV_RUNTIME_ENV_FLAG] = "0"
+    try:
+        yield
+    finally:
+        if previous_ray_flag is None:
+            os.environ.pop(_RAY_UV_RUNTIME_ENV_FLAG, None)
+        else:
+            os.environ[_RAY_UV_RUNTIME_ENV_FLAG] = previous_ray_flag
+        os.environ.update(previous)
+
+
+def disable_ray_uv_runtime_env_hook(ray: object) -> None:
+    """Disable Ray's parent-process uv hook when Ray was imported earlier."""
+
+    private = getattr(ray, "_private", None)
+    constants = getattr(private, "ray_constants", None)
+    if constants is not None:
+        constants.RAY_ENABLE_UV_RUN_RUNTIME_ENV = False
+
+
+def build_local_ray_runtime_env() -> dict[str, Any]:
+    """Build the runtime env that lets local Ray workers reuse this process environment."""
+
+    venv = os.path.dirname(os.path.dirname(sys.executable))
+    venv_bin = os.path.join(venv, "bin")
+    pypath = os.pathsep.join(p for p in sys.path if p)
+    ray_env_vars: dict[str, str] = {
+        "VIRTUAL_ENV": venv,
+        "PATH": venv_bin + os.pathsep + os.environ.get("PATH", ""),
+        "PYTHONPATH": pypath,
+    }
+    ray_env_vars.update(collect_hf_runtime_env())
+    ray_env_vars.update(collect_remote_auth_runtime_env())
+    if "HF_HUB_OFFLINE" in ray_env_vars:
+        os.environ["HF_HUB_OFFLINE"] = ray_env_vars["HF_HUB_OFFLINE"]
+    return {"env_vars": ray_env_vars, "py_executable": sys.executable}
+
+
+def ensure_local_ray_runtime(ray_address: str | None = None, *, log_to_driver: bool | None = None) -> object:
+    """Import Ray and initialize it with Retriever's local worker runtime env."""
+
+    with without_uv_run_env():
+        import ray
+
+        disable_ray_uv_runtime_env_hook(ray)
+        if ray_address or not ray.is_initialized():
+            init_kwargs: dict[str, Any] = {
+                "address": ray_address,
+                "ignore_reinit_error": True,
+                "runtime_env": build_local_ray_runtime_env(),
+            }
+            if log_to_driver is not None:
+                init_kwargs["log_to_driver"] = log_to_driver
+            ray.init(**init_kwargs)
+        return ray

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/examples/graph_pipeline.py

@@ -9,8 +9,7 @@ as the ``retriever pipeline run`` CLI subcommand.
 
 This module re-exports the same Typer :data:`app` and keeps the
 ``python -m nemo_retriever.examples.graph_pipeline <args>`` entry point
-working so existing callers (notably
-:mod:`nemo_retriever.harness.run`) do not need to change.
+working for existing pipeline callers.
 
 New code should invoke the pipeline via one of the following:
 

{nemo_retriever-2026.6.28.dev136 → nemo_retriever-2026.6.29.dev137}/src/nemo_retriever/graph/executor.py

@@ -6,8 +6,6 @@
 
 from __future__ import annotations
 
-import os
-import sys
 from abc import ABC, abstractmethod
 from typing import Any, Dict, List, Optional
 import pandas as pd
@@ -15,13 +13,12 @@ import pandas as pd
 from nemo_retriever.operators.gpu_operator import GPUOperator
 from nemo_retriever.graph.pipeline_graph import Graph, Node
 from nemo_retriever.graph.operator_resolution import resolve_graph
-from nemo_retriever.models.hf_cache import collect_hf_runtime_env
 from nemo_retriever.common.input_files import (
     _is_explicit_glob_path,
     expand_input_file_patterns,
     raise_input_path_not_found,
 )
-from nemo_retriever.common.remote_auth import collect_remote_auth_runtime_env
+from nemo_retriever.common.ray_runtime import ensure_local_ray_runtime
 from nemo_retriever.common import ray_resource_hueristics as _rrh
 from nemo_retriever.common.ray_resource_hueristics import (
     gather_cluster_resources,
@@ -231,7 +228,7 @@ class RayDataExecutor(AbstractExecutor):
         ray.data.Dataset
             The lazy Ray dataset with all graph stages appended.
         """
-        import ray
+        ray = ensure_local_ray_runtime(self._ray_address)
         import ray.data as rd
 
         if not isinstance(data, (rd.Dataset, str, list)):
@@ -243,25 +240,6 @@ class RayDataExecutor(AbstractExecutor):
         if isinstance(data, (str, list)):
             input_paths = expand_input_file_patterns(data)
 
-        if self._ray_address or not ray.is_initialized():
-            venv = os.path.dirname(os.path.dirname(sys.executable))
-            venv_bin = os.path.join(venv, "bin")
-            pypath = os.pathsep.join(p for p in sys.path if p)
-            ray_env_vars: dict[str, str] = {
-                "VIRTUAL_ENV": venv,
-                "PATH": venv_bin + os.pathsep + os.environ.get("PATH", ""),
-                "PYTHONPATH": pypath,
-            }
-            ray_env_vars.update(collect_hf_runtime_env())
-            ray_env_vars.update(collect_remote_auth_runtime_env())
-            os.environ["HF_HUB_OFFLINE"] = ray_env_vars["HF_HUB_OFFLINE"]
-            runtime_env = {"env_vars": ray_env_vars}
-            ray.init(
-                address=self._ray_address,
-                ignore_reinit_error=True,
-                runtime_env=runtime_env,
-            )
-
         ctx = rd.DataContext.get_current()
         ctx.enable_rich_progress_bars = True
         ctx.use_ray_tqdm = False