verifiers 0.1.15.dev168__tar.gz → 0.1.15.dev169__tar.gz

{verifiers-0.1.15.dev168 → verifiers-0.1.15.dev169}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: verifiers
-Version: 0.1.15.dev168
+Version: 0.1.15.dev169
 Summary: Verifiers: Environments for LLM Reinforcement Learning
 Project-URL: Homepage, https://github.com/primeintellect-ai/verifiers
 Project-URL: Documentation, https://github.com/primeintellect-ai/verifiers
@@ -22,8 +22,10 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: <3.14,>=3.10
+Requires-Dist: aiohttp>=3.9.0
 Requires-Dist: aiolimiter>=1.2.1
 Requires-Dist: anthropic>=0.78.0
+Requires-Dist: certifi
 Requires-Dist: datasets<4.7.0,>=3.0.0
 Requires-Dist: gepa
 Requires-Dist: httpx>=0.27.0
@@ -35,10 +37,12 @@ Requires-Dist: nest-asyncio>=1.6.0
 Requires-Dist: numpy
 Requires-Dist: openai-agents>=0.0.7
 Requires-Dist: openai>=1.108.1
+Requires-Dist: pillow
 Requires-Dist: prime-pydantic-config[toml]
 Requires-Dist: prime-sandboxes>=0.2.25
 Requires-Dist: prime-tunnel>=0.1.6
 Requires-Dist: pydantic>=2.11.9
+Requires-Dist: pymupdf
 Requires-Dist: pyzmq>=27.1.0
 Requires-Dist: regex<2026.4.4
 Requires-Dist: requests

{verifiers-0.1.15.dev168 → verifiers-0.1.15.dev169}/pyproject.toml

@@ -53,6 +53,10 @@ dependencies = [
     "setproctitle>=1.3.0",
     "regex<2026.4.4",
     "httpx>=0.27.0",
+    "aiohttp>=3.9.0",
+    "pymupdf",
+    "pillow",
+    "certifi",
     "prime-pydantic-config[toml]",
     "uvloop>=0.21.0; sys_platform != 'win32' and sys_platform != 'cygwin' and platform_python_implementation != 'PyPy'",
 ]

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/README.md

@@ -0,0 +1,35 @@
+# Search Tasksets
+
+Composable search/research tasksets for agents that solve live information-seeking tasks in a sandbox.
+
+The search family is intentionally backend-oriented, mirroring the SWE taskset pattern while keeping the task contract research-centric: each task expects a single final answer rather than a code patch. Agents may use web/search tools, browser helpers, or other sandbox resources provided by the paired environment.
+
+## Backends
+
+| Backend | Source | Default dataset | Status |
+|---|---|---|---|
+| `quest` | [OSU-NLP-Group/QUEST](https://github.com/OSU-NLP-Group/QUEST) | [`osunlp/QUEST-RL-Data`](https://huggingface.co/datasets/osunlp/QUEST-RL-Data) | Objective tasks supported |
+
+## Usage
+
+```python
+from verifiers.envs.experimental.composable.tasksets.search import make_search_taskset
+
+taskset = make_search_taskset(backend="quest", category="objective")
+```
+
+`make_search_taskset()` dispatches by backend name. Unknown backends raise `ValueError` with the available backend list.
+
+## Output Contract
+
+Search tasksets should define their own output contract. The initial `quest` backend expects the agent to write one final researched response to `/task/answer.txt`, including supporting URLs/citations when available. Scratch reasoning, tool traces, and logs should not be written as the final answer.
+
+## Error Handling
+
+Search tasksets should use the framework error taxonomy for infrastructure failures:
+
+- `vf.SandboxError` for sandbox setup, command, or lifecycle failures.
+- `vf.ModelError` for judge/model provider failures.
+- `vf.InfraError` for dataset, evaluator, or external runtime failures.
+
+Incorrect answers should not set `state["error"]`; they should score normally, often as `0.0`.

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/__init__.py

@@ -0,0 +1,5 @@
+"""Composable search/research tasksets."""
+
+from .search_tasksets import make_quest_taskset, make_search_taskset
+
+__all__ = ["make_quest_taskset", "make_search_taskset"]

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/quest/README.md

@@ -0,0 +1,52 @@
+# QUEST Search Taskset
+
+Objective QUEST tasks ported into the composable search taskset framework.
+
+## Source
+
+- Dataset: [`osunlp/QUEST-RL-Data`](https://huggingface.co/datasets/osunlp/QUEST-RL-Data)
+- Upstream project: [`OSU-NLP-Group/QUEST`](https://github.com/OSU-NLP-Group/QUEST)
+
+The taskset loads the Hugging Face dataset, filters to `rl_task_category == "objective"` by default, and uses the dataset-provided generated evaluation scripts under `eval_scripts/*.py`.
+
+## Task Contract
+
+Each example is a live research question. The agent should produce one final answer in `/task/answer.txt`.
+
+The paired `rlm_search` environment prompts RLM to write this file and provides web search/open-page skills. The rubric can fall back to the final assistant text if the answer file is empty, but agents should still write the file directly.
+
+## Scoring
+
+`QuestRubric` loads the generated eval script for the example's `task_id` and calls its async `evaluate_answer(...)` entrypoint using the vendored minimal `obj_task_eval` runtime. The rollout reward is `summary["final_score"]`, clipped to `[0.0, 1.0]`.
+
+Generated scripts may request URL-backed verification. PDF URLs are detected and parsed with the upstream QUEST PDF parser path before falling back to generic webpage retrieval.
+
+This port intentionally preserves upstream QUEST behavior for URL-backed verification semantics. The upstream verifier generally treats invalid, irrelevant, or inaccessible cited webpages as unsupported claims, which can assign `0.0` to the affected verification node even when the immediate cause is source access such as a bot challenge, rate limit, timeout, or parser failure. Future work should consider a finer-grained source-access taxonomy so verifier infrastructure limitations can be distinguished from model-provided bad URLs or unsupported claims.
+
+A reward of `0.0` with no `state["error"]` means the QUEST evaluator ran and judged the answer incorrect under the upstream-compatible scoring path. Infrastructure and evaluator failures outside normal QUEST source verification are represented with `vf.Error` subclasses instead of ad hoc success metrics.
+
+## Error Handling
+
+QUEST uses Verifiers' framework-managed error field for non-answer failures when the failure comes from external runtime systems:
+
+- Missing live sandbox or answer-file read failure: `vf.SandboxError`.
+- Transient judge provider/network/rate-limit/server failures: retryable `vf.InfraError`.
+- Empty or invalid judge responses: retryable `vf.InvalidModelResponseError` / `vf.EmptyModelResponseError`.
+- Judge auth, model-not-found, content-filter, or invalid request failures: non-retryable `vf.ModelError`.
+- QUEST eval-script download/cache resolution failure: `vf.InfraError`.
+
+Wrong answers, empty answers, and inaccessible or irrelevant cited sources remain ordinary scored outcomes and return `0.0` without setting `state["error"]`. Generated eval-script source errors, missing task metadata, missing eval-script files, import/load failures, and unexpected evaluator runtime bugs are not converted to `vf.Error`; they raise normally so broken evaluator code fails hard.
+
+## Common Arguments
+
+| Argument | Default | Description |
+|---|---:|---|
+| `dataset_name` | `osunlp/QUEST-RL-Data` | Hugging Face dataset name. |
+| `split` | `train` | Dataset split. |
+| `category` | `objective` | Initial implementation supports objective tasks only. |
+| `answer_file` | `/task/answer.txt` | Final answer path in the sandbox. |
+| `judge_model` | `openai/gpt-5.4-mini` | OpenAI-compatible model for QUEST verifier calls. |
+| `judge_base_url` | `https://api.pinference.ai/api/v1` | Judge API base URL. |
+| `judge_api_key_var` | `PRIME_API_KEY` | Env var containing the judge API key. |
+| `quest_eval_scripts_dir` | HF cache | Optional local directory containing `eval_scripts/*.py`. |
+| `quest_cache_dir` | `~/.cache/verifiers/quest` | Host cache for QUEST verifier state. |

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/quest/__init__.py

@@ -0,0 +1,5 @@
+"""QUEST search taskset."""
+
+from .taskset import QuestRubric, QuestTaskSet
+
+__all__ = ["QuestRubric", "QuestTaskSet"]

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/quest/obj_task_eval/__init__.py

@@ -0,0 +1,17 @@
+"""Vendored QUEST objective evaluation runtime."""
+
+from .eval_toolkit import BinaryEvalResult, Extractor, Verifier, create_evaluator
+from .evaluator import Evaluator
+from .utils import CacheFileSys
+from .verification_tree import AggregationStrategy, VerificationNode
+
+__all__ = [
+    "AggregationStrategy",
+    "BinaryEvalResult",
+    "CacheFileSys",
+    "Evaluator",
+    "Extractor",
+    "Verifier",
+    "VerificationNode",
+    "create_evaluator",
+]

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/quest/obj_task_eval/api_tools/__init__.py

@@ -0,0 +1,5 @@
+"""Vendored QUEST API tool shims."""
+
+from .tool_pdf import PDFParser
+
+__all__ = ["PDFParser"]

verifiers-0.1.15.dev169/verifiers/envs/experimental/composable/tasksets/search/quest/obj_task_eval/api_tools/tool_pdf.py

@@ -0,0 +1,275 @@
+"""Lightweight PDF parser from QUEST objective evaluation.
+
+The parser accepts a URL, local path, bytes, or ``BytesIO`` object and returns
+``(imgs, text)``. ``imgs`` is a list of base64-encoded page JPEGs and ``text``
+is extracted page text. Failures return a blank image plus an explanatory text
+message, matching upstream QUEST's tolerant evaluator behavior.
+"""
+
+import asyncio
+import base64
+import random
+from io import BytesIO
+from logging import Logger
+from typing import List, Optional, Tuple, Union
+from urllib.parse import unquote, urlparse
+
+import aiohttp
+import certifi
+import fitz
+import httpx
+import requests
+from PIL import Image
+
+from ..utils.url_tools import normalize_url_for_browser
+
+PDF_MAGIC = b"%PDF-"
+UA_CHROME = (
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+    "AppleWebKit/537.36 (KHTML, like Gecko) "
+    "Chrome/124.0.0.0 Safari/537.36"
+)
+USER_AGENT_STRINGS = [
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 OPR/109.0.0.0",
+]
+
+
+def make_blank_png_b64() -> str:
+    """Return a transparent 1x1 PNG as base64."""
+    img = Image.new("RGBA", (1, 1), (0, 0, 0, 0))
+    buf = BytesIO()
+    img.save(buf, format="PNG")
+    return base64.b64encode(buf.getvalue()).decode()
+
+
+def is_pdf_by_suffix(url: str) -> bool:
+    """Check whether a URL likely points to a PDF based on path/query patterns."""
+    parsed = urlparse(url.lower())
+    path = unquote(parsed.path)
+    if path.endswith(".pdf"):
+        return True
+
+    pdf_patterns = [
+        "arxiv.org/pdf/",
+        "/download/pdf",
+        "/fulltext.pdf",
+        "/article/pdf",
+        "/content/pdf",
+        "type=pdf",
+        "format=pdf",
+        "download=pdf",
+        ".pdf?",
+        "/pdf/",
+        "pdfviewer",
+    ]
+    url_lower = url.lower()
+    return any(pattern in url_lower for pattern in pdf_patterns)
+
+
+def is_pdf_by_requests_head(url: str) -> bool:
+    """Check via HEAD request whether a URL is a PDF."""
+    try:
+        response = requests.head(
+            url,
+            allow_redirects=True,
+            timeout=10,
+            verify=certifi.where(),
+        )
+        content_type = response.headers.get("content-type", "").lower()
+        return "pdf" in content_type
+    except requests.RequestException:
+        return False
+
+
+async def is_pdf_by_httpx_get_range(url: str, timeout: int = 10) -> bool:
+    """Check PDF via a partial GET request to read the file header."""
+    try:
+        async with httpx.AsyncClient(
+            follow_redirects=True,
+            timeout=timeout,
+            verify=False,
+        ) as client:
+            headers = {
+                "User-Agent": random.choice(USER_AGENT_STRINGS),
+                "Range": "bytes=0-1023",
+                "Accept": "*/*",
+            }
+            response = await client.get(url, headers=headers)
+            content_type = (
+                response.headers.get("content-type", "").split(";")[0].strip().lower()
+            )
+            if "pdf" in content_type:
+                return True
+            return bool(response.content and response.content.startswith(PDF_MAGIC))
+    except (httpx.TimeoutException, httpx.ConnectError, httpx.HTTPError):
+        return False
+    except Exception:
+        return False
+
+
+async def is_pdf_by_full_get(url: str, timeout: int = 15) -> bool:
+    """Last-resort PDF detection by streaming the start of the response body."""
+    try:
+        async with httpx.AsyncClient(
+            follow_redirects=True,
+            timeout=timeout,
+            verify=False,
+        ) as client:
+            headers = {
+                "User-Agent": random.choice(USER_AGENT_STRINGS),
+                "Accept": "*/*",
+            }
+            async with client.stream("GET", url, headers=headers) as response:
+                chunk_data = b""
+                async for chunk in response.aiter_bytes(chunk_size=5):
+                    chunk_data += chunk
+                    if len(chunk_data) >= 5:
+                        break
+                if chunk_data and chunk_data.startswith(PDF_MAGIC):
+                    return True
+                content_type = (
+                    response.headers.get("content-type", "")
+                    .split(";")[0]
+                    .strip()
+                    .lower()
+                )
+                return "pdf" in content_type
+    except Exception:
+        return False
+
+
+async def is_pdf(url: str, logger: Optional[Logger] = None) -> bool:
+    """Robustly detect whether a URL points to a PDF file."""
+    url = normalize_url_for_browser(url)
+    if logger:
+        logger.debug(f"Checking if URL is PDF: {url}")
+
+    if is_pdf_by_suffix(url):
+        if logger:
+            logger.info(f"URL pattern indicates PDF: {url}")
+        return True
+    if is_pdf_by_requests_head(url):
+        if logger:
+            logger.info(f"HEAD request confirms PDF: {url}")
+        return True
+    if await is_pdf_by_httpx_get_range(url):
+        if logger:
+            logger.info(f"Partial GET confirms PDF: {url}")
+        return True
+    if await is_pdf_by_full_get(url):
+        if logger:
+            logger.info(f"Full GET confirms PDF: {url}")
+        return True
+    return False
+
+
+class PDFParser:
+    """Download and parse PDFs for QUEST source verification."""
+
+    MAX_PAGES: int = 100
+    MAX_IMAGE_PAGES: int = 50
+    RENDER_DPI: int = 144
+    JPEG_QUALITY: int = 70
+
+    async def extract(
+        self,
+        source: Union[str, bytes, BytesIO],
+    ) -> Tuple[Optional[List[str]], Optional[str]]:
+        """Extract page screenshots and text from a PDF source."""
+        try:
+            if isinstance(source, (bytes, BytesIO)):
+                data = source.getvalue() if isinstance(source, BytesIO) else source
+            elif isinstance(source, str) and source.lower().startswith(
+                ("http://", "https://")
+            ):
+                data = await self._fetch_pdf_bytes(source)
+            else:
+                data = await asyncio.to_thread(
+                    lambda path: open(path, "rb").read(), str(source)
+                )
+
+            if not data.lstrip().startswith(PDF_MAGIC):
+                return [
+                    make_blank_png_b64()
+                ], "PDF extraction failed: Invalid PDF format"
+
+            return await asyncio.to_thread(self._extract_from_bytes, data)
+        except Exception as exc:
+            return [make_blank_png_b64()], f"PDF extraction failed: {exc}"
+
+    async def parse_url(self, url: str) -> str | None:
+        """Compatibility helper returning text for a PDF URL."""
+        _imgs, text = await self.extract(url)
+        return text
+
+    def parse_bytes(self, data: bytes) -> str | None:
+        """Compatibility helper returning text for PDF bytes."""
+        _imgs, text = self._extract_from_bytes(data)
+        return text
+
+    async def _fetch_pdf_bytes(self, url: str) -> bytes:
+        """Fetch PDF bytes with a browser user agent and an arXiv backup domain."""
+        headers = {
+            "User-Agent": UA_CHROME,
+            "Accept": "application/pdf,application/octet-stream;q=0.9,*/*;q=0.8",
+        }
+
+        async def download(fetch_url: str) -> bytes:
+            async with aiohttp.ClientSession(headers=headers) as session:
+                async with session.get(
+                    fetch_url, allow_redirects=True, timeout=30
+                ) as response:
+                    response.raise_for_status()
+                    return await response.read()
+
+        data = await download(url)
+        if not data.lstrip().startswith(PDF_MAGIC) and "arxiv.org" in url:
+            backup = url.replace("://arxiv.org", "://export.arxiv.org")
+            try:
+                data = await download(backup)
+            except Exception:
+                pass
+        return data
+
+    def _extract_from_bytes(
+        self, data: bytes
+    ) -> Tuple[Optional[List[str]], Optional[str]]:
+        """Parse PDF bytes into page images and text."""
+        if not data.lstrip().startswith(PDF_MAGIC):
+            return [make_blank_png_b64()], "PDF extraction failed: Invalid PDF format"
+
+        try:
+            doc = fitz.open(stream=data, filetype="pdf")
+        except (fitz.FileDataError, RuntimeError):
+            return [
+                make_blank_png_b64()
+            ], "PDF extraction failed: Unable to parse PDF file"
+
+        imgs: List[str] = []
+        texts: List[str] = []
+        zoom = self.RENDER_DPI / 72
+        max_pages = min(self.MAX_PAGES, doc.page_count)
+        max_img_pages = min(self.MAX_IMAGE_PAGES, doc.page_count)
+
+        for index in range(max_pages):
+            page = doc.load_page(index)
+            texts.append(page.get_text("text"))
+            if index < max_img_pages:
+                pix = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom), alpha=False)
+                img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
+                buf = BytesIO()
+                img.save(
+                    buf,
+                    "JPEG",
+                    quality=self.JPEG_QUALITY,
+                    optimize=True,
+                    progressive=True,
+                )
+                imgs.append(base64.b64encode(buf.getvalue()).decode())
+        doc.close()
+        return imgs, "\n".join(texts)