opensolr-haystack 0.6.0__tar.gz → 0.7.0__tar.gz

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opensolr-haystack
-Version: 0.6.0
+Version: 0.7.0
 Summary: Haystack integration for Opensolr — managed Apache Solr DocumentStore with server-side embeddings and hybrid BM25+kNN retrieval
 Author-email: Opensolr <support@opensolr.com>
 License: MIT
@@ -61,8 +61,8 @@ the query. The store embeds server-side at both index and query time.
 
 ## Try it without an account
 
-There is a public demo account. Point the package at it and everything in this
-README works immediately, with no signup:
+There is a public demo account. Point the package at it and search, ingestion and
+answers work immediately, with no signup:
 
 ```bash
 export OPENSOLR_EMAIL=mcp@opensolr.com
@@ -72,6 +72,8 @@ export OPENSOLR_API_KEY=420b8b23e7b12dc8ab838932145a5065
 `mcp_demo_d1__dense` is already loaded with 300 news articles, so search, filtering
 and grounded answers work the moment you connect. You also get the full write path:
 create your own index on the account, ingest into it, query it, delete it.
+Reading files and pictures (the file loader and file indexing below, search by image)
+needs your own key: the shared demo key does not include them.
 
 Know what you are working with:
 
@@ -198,6 +200,63 @@ docs = store.search_by_image("photo.jpg", using="all")      # labels + OCR + cod
 `alpha`, `fresh_bias`, `filters` — so an image query runs through the exact
 hybrid pipeline a text query does.
 
+## Local files: PDFs page by page, scans through OCR
+
+`OpensolrConverter` turns files into Haystack `Document`s — one per PDF page
+(`meta["page_number"]`), one per any other file. It takes file paths, folders and
+`ByteStream`s:
+
+```python
+from haystack_integrations.components.converters.opensolr import OpensolrConverter
+
+converter = OpensolrConverter(document_store=store)
+docs = converter.run(sources=["./contracts", "scan.png"], meta={"team": "legal"})["documents"]
+# meta: file_path, page_number, is_ocr, language, content_type (+ yours)
+```
+
+Or index them straight into the store — no converter, no splitter, PDFs page by page:
+
+```python
+result = store.write_files("./contracts", pattern="*.pdf", metadata={"team": "legal"})
+# {"documents": [{path, source, uri, id}], "pending": [], "failed": [], "jobs": [...]}
+```
+
+- Each file is read on Opensolr's servers: its text, the text in its pictures (OCR,
+  more than 100 languages), a short description of the pictures without text. Word,
+  RTF, OpenDocument, text, HTML and picture files (scans) are one document each; a
+  **PDF is one document per page**.
+- The bytes travel once: a file the account already sent (same content) is not
+  uploaded again. Pictures read by OCR count 0.5 of an AI request each; a file read
+  before is free.
+- Files still being read when `timeout` runs out are under `result["pending"]` (the
+  converter logs them): run again to get them, nothing is sent twice.
+- An indexed file's `uri` is `https://ingest.opensolr.com/<index>/<source>`, where
+  `source` is its path from the folder you sent (`meta["source"]`); sending it again
+  replaces it. Its date is the file's modification time.
+
+## PDF pages in search results
+
+On an index where PDFs are stored page by page (files indexed as above, PDFs sent to
+Data Ingestion with `rtf`, PDFs found by the Web Crawler), a search returns each PDF
+**once**, as its best page. Its metadata carries `page_number` (the page number),
+`page_url` (opens the PDF at that page, `...pdf#page=14`), `document_id`,
+`more_pages` (its other matching pages, each with `page` and `page_url`) and
+`matching_pages`; every hit carries its `uri`. `group_pages=False` returns the pages
+as separate results.
+
+## Searching a period
+
+`date_from` and `date_to` (`YYYY-MM-DD`, both included, either one can be left out)
+keep only the documents dated in that period (`creation_date`):
+
+```python
+docs = store.search("rate decision", top_k=5, date_from="2026-01-01", date_to="2026-03-31")
+# or in a pipeline:
+OpensolrHybridRetriever(document_store=store, date_from="2026-01-01")
+```
+
+They combine with every other option — filters, `fresh_bias`, hybrid or lexical.
+
 ## Your index schema
 
 Documents follow the Opensolr document model (`title`, `description`, `text`,

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/README.md

@@ -43,8 +43,8 @@ the query. The store embeds server-side at both index and query time.
 
 ## Try it without an account
 
-There is a public demo account. Point the package at it and everything in this
-README works immediately, with no signup:
+There is a public demo account. Point the package at it and search, ingestion and
+answers work immediately, with no signup:
 
 ```bash
 export OPENSOLR_EMAIL=mcp@opensolr.com
@@ -54,6 +54,8 @@ export OPENSOLR_API_KEY=420b8b23e7b12dc8ab838932145a5065
 `mcp_demo_d1__dense` is already loaded with 300 news articles, so search, filtering
 and grounded answers work the moment you connect. You also get the full write path:
 create your own index on the account, ingest into it, query it, delete it.
+Reading files and pictures (the file loader and file indexing below, search by image)
+needs your own key: the shared demo key does not include them.
 
 Know what you are working with:
 
@@ -180,6 +182,63 @@ docs = store.search_by_image("photo.jpg", using="all")      # labels + OCR + cod
 `alpha`, `fresh_bias`, `filters` — so an image query runs through the exact
 hybrid pipeline a text query does.
 
+## Local files: PDFs page by page, scans through OCR
+
+`OpensolrConverter` turns files into Haystack `Document`s — one per PDF page
+(`meta["page_number"]`), one per any other file. It takes file paths, folders and
+`ByteStream`s:
+
+```python
+from haystack_integrations.components.converters.opensolr import OpensolrConverter
+
+converter = OpensolrConverter(document_store=store)
+docs = converter.run(sources=["./contracts", "scan.png"], meta={"team": "legal"})["documents"]
+# meta: file_path, page_number, is_ocr, language, content_type (+ yours)
+```
+
+Or index them straight into the store — no converter, no splitter, PDFs page by page:
+
+```python
+result = store.write_files("./contracts", pattern="*.pdf", metadata={"team": "legal"})
+# {"documents": [{path, source, uri, id}], "pending": [], "failed": [], "jobs": [...]}
+```
+
+- Each file is read on Opensolr's servers: its text, the text in its pictures (OCR,
+  more than 100 languages), a short description of the pictures without text. Word,
+  RTF, OpenDocument, text, HTML and picture files (scans) are one document each; a
+  **PDF is one document per page**.
+- The bytes travel once: a file the account already sent (same content) is not
+  uploaded again. Pictures read by OCR count 0.5 of an AI request each; a file read
+  before is free.
+- Files still being read when `timeout` runs out are under `result["pending"]` (the
+  converter logs them): run again to get them, nothing is sent twice.
+- An indexed file's `uri` is `https://ingest.opensolr.com/<index>/<source>`, where
+  `source` is its path from the folder you sent (`meta["source"]`); sending it again
+  replaces it. Its date is the file's modification time.
+
+## PDF pages in search results
+
+On an index where PDFs are stored page by page (files indexed as above, PDFs sent to
+Data Ingestion with `rtf`, PDFs found by the Web Crawler), a search returns each PDF
+**once**, as its best page. Its metadata carries `page_number` (the page number),
+`page_url` (opens the PDF at that page, `...pdf#page=14`), `document_id`,
+`more_pages` (its other matching pages, each with `page` and `page_url`) and
+`matching_pages`; every hit carries its `uri`. `group_pages=False` returns the pages
+as separate results.
+
+## Searching a period
+
+`date_from` and `date_to` (`YYYY-MM-DD`, both included, either one can be left out)
+keep only the documents dated in that period (`creation_date`):
+
+```python
+docs = store.search("rate decision", top_k=5, date_from="2026-01-01", date_to="2026-03-31")
+# or in a pipeline:
+OpensolrHybridRetriever(document_store=store, date_from="2026-01-01")
+```
+
+They combine with every other option — filters, `fresh_bias`, hybrid or lexical.
+
 ## Your index schema
 
 Documents follow the Opensolr document model (`title`, `description`, `text`,

opensolr_haystack-0.7.0/haystack_integrations/components/converters/opensolr/__init__.py

@@ -0,0 +1,5 @@
+"""Opensolr converter component for Haystack: files read on Opensolr, a PDF page by page."""
+
+from haystack_integrations.components.converters.opensolr.converter import OpensolrConverter
+
+__all__ = ["OpensolrConverter"]

opensolr_haystack-0.7.0/haystack_integrations/components/converters/opensolr/converter.py

@@ -0,0 +1,123 @@
+"""Converter component: local files read on Opensolr's servers (text, OCR, a PDF page by page)."""
+
+from __future__ import annotations
+
+import logging
+import os
+import tempfile
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+from haystack import Document, component, default_from_dict, default_to_dict
+from haystack.dataclasses import ByteStream
+
+from haystack_integrations.document_stores.opensolr import OpensolrDocumentStore
+from haystack_integrations.document_stores.opensolr.client import list_files
+
+logger = logging.getLogger(__name__)
+
+
+@component
+class OpensolrConverter:
+    """Turn files into Documents, read on Opensolr's servers.
+
+    Each file is read there: its text, the text in its pictures (OCR, more than 100
+    languages), a short description of the pictures that have no text, scanned pages
+    and picture files (JPG, PNG, TIFF...) included. A PDF gives one Document per page
+    (``meta["page_number"]``), any other file one Document. Nothing is indexed: put an
+    embedder and a writer after it, or use ``OpensolrDocumentStore.write_files`` to
+    index files directly.
+
+    Example:
+        ```python
+        from haystack_integrations.document_stores.opensolr import OpensolrDocumentStore
+        from haystack_integrations.components.converters.opensolr import OpensolrConverter
+
+        store = OpensolrDocumentStore(index="mysite__dense")
+        converter = OpensolrConverter(document_store=store)
+        docs = converter.run(sources=["./contracts"])["documents"]
+        ```
+
+    The work is counted on the store's index: each picture read costs 0.5 of an AI
+    request; a file this account sent before is read for free. ``sources`` takes file
+    paths, folders (every file in them, sub-folders included) and ``ByteStream``s.
+    Files that could not be read, or were still being read when ``timeout`` ran out,
+    are logged and left out; run again to get the late ones (they are not sent twice).
+    """
+
+    #: Files sent, and waited for, together.
+    BATCH = 20
+
+    def __init__(self, document_store: OpensolrDocumentStore, timeout: float = 600.0) -> None:
+        self.document_store = document_store
+        self.timeout = timeout
+
+    @component.output_types(documents=List[Document])
+    def run(
+        self,
+        sources: List[Union[str, Path, ByteStream]],
+        meta: Optional[Union[Dict[str, Any], List[Dict[str, Any]]]] = None,
+    ) -> Dict[str, List[Document]]:
+        """``meta``: added to every Document, one dict for all sources or one per source."""
+        if isinstance(meta, list) and len(meta) != len(sources):
+            raise ValueError("meta must be one dict, or one dict per source")
+        jobs: List[tuple] = []
+        temps: List[str] = []
+        try:
+            for i, source in enumerate(sources):
+                extra = dict((meta[i] if isinstance(meta, list) else meta) or {})
+                if isinstance(source, ByteStream):
+                    name = Path(str(source.meta.get("file_path") or "document")).name
+                    fd, tmp = tempfile.mkstemp(suffix="-" + name)
+                    with os.fdopen(fd, "wb") as fh:
+                        fh.write(source.data)
+                    temps.append(tmp)
+                    jobs.append((tmp, {**(source.meta or {}), "file_path": source.meta.get("file_path") or name, **extra}))
+                else:
+                    for f in list_files(source):
+                        jobs.append((str(f), {"file_path": str(f), **extra}))
+            documents: List[Document] = []
+            client = self.document_store.client
+            for i in range(0, len(jobs), self.BATCH):
+                batch = jobs[i : i + self.BATCH]
+                entries = client.extract_files(self.document_store.index, [p for p, _ in batch], pages=True, timeout=self.timeout)
+                for entry, (_, base) in zip(entries, batch):
+                    documents.extend(self._documents(entry, base))
+            return {"documents": documents}
+        finally:
+            for tmp in temps:
+                try:
+                    os.remove(tmp)
+                except OSError:
+                    pass
+
+    @staticmethod
+    def _documents(entry: Dict[str, Any], base: Dict[str, Any]) -> List[Document]:
+        state = entry.get("state")
+        if state != "done":
+            logger.warning("Opensolr did not read %s (%s): %s", base.get("file_path"), state, entry.get("error") or "")
+            return []
+        base = {**base, "content_type": entry.get("mime") or ""}
+        pages = entry.get("pages") or []
+        if pages:
+            return [
+                Document(content=str(p.get("text") or ""), meta={
+                    **base,
+                    "page_number": int(p.get("page") or 0),
+                    "is_ocr": bool(p.get("is_ocr")),
+                    "language": str(p.get("lang") or ""),
+                })
+                for p in pages if str(p.get("text") or "").strip()
+            ]
+        text = str(entry.get("text") or "")
+        return [Document(content=text, meta=base)] if text.strip() else []
+
+    def to_dict(self) -> Dict[str, Any]:
+        return default_to_dict(self, document_store=self.document_store.to_dict(), timeout=self.timeout)
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "OpensolrConverter":
+        data["init_parameters"]["document_store"] = OpensolrDocumentStore.from_dict(
+            data["init_parameters"]["document_store"]
+        )
+        return default_from_dict(cls, data)

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/haystack_integrations/components/retrievers/opensolr/retriever.py

@@ -39,6 +39,9 @@ class OpensolrHybridRetriever:
         filters: Optional[Dict[str, Any]] = None,
         lexical: bool = False,
         fresh_bias: bool = False,
+        date_from: Optional[str] = None,
+        date_to: Optional[str] = None,
+        group_pages: bool = True,
     ) -> None:
         self.document_store = document_store
         self.top_k = top_k
@@ -51,6 +54,9 @@ class OpensolrHybridRetriever:
         # for a Haystack user, so an option the component does not carry is an option
         # that does not exist in a pipeline.
         self.fresh_bias = fresh_bias
+        self.date_from = date_from
+        self.date_to = date_to
+        self.group_pages = group_pages
 
     @component.output_types(documents=List[Document])
     def run(
@@ -62,13 +68,19 @@ class OpensolrHybridRetriever:
         filters: Optional[Dict[str, Any]] = None,
         lexical: Optional[bool] = None,
         fresh_bias: Optional[bool] = None,
+        date_from: Optional[str] = None,
+        date_to: Optional[str] = None,
+        group_pages: Optional[bool] = None,
     ) -> Dict[str, List[Document]]:
         """Run the retriever. ``alpha``: 0 = all semantic, 1 = all lexical.
         ``lexical=True`` = pure keyword search, no embedding call.
         ``fresh_bias=True`` biases the ranking toward recent documents by
         multiplying each score by a recency curve on ``creation_date``; it
         re-orders and never filters, so the hit count is unchanged and nothing
-        old becomes unreachable. Off by default."""
+        old becomes unreachable. Off by default.
+        ``date_from`` / ``date_to`` (``YYYY-MM-DD``) keep only documents dated in that
+        period. ``group_pages`` (on by default) returns a PDF indexed page by page once,
+        as its best page, with ``page_number``, ``page_url`` and ``more_pages`` in meta."""
         docs = self.document_store.search(
             query=query,
             top_k=top_k if top_k is not None else self.top_k,
@@ -77,6 +89,9 @@ class OpensolrHybridRetriever:
             filters=filters if filters is not None else self.filters,
             lexical=lexical if lexical is not None else self.lexical,
             fresh_bias=fresh_bias if fresh_bias is not None else self.fresh_bias,
+            date_from=date_from if date_from is not None else self.date_from,
+            date_to=date_to if date_to is not None else self.date_to,
+            group_pages=group_pages if group_pages is not None else self.group_pages,
         )
         return {"documents": docs}
 
@@ -93,6 +108,9 @@ class OpensolrHybridRetriever:
             # the bias on comes back with it on. from_dict() tolerates its absence in
             # dicts written before this option existed — the constructor defaults it.
             fresh_bias=self.fresh_bias,
+            date_from=self.date_from,
+            date_to=self.date_to,
+            group_pages=self.group_pages,
         )
 
     @classmethod

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/haystack_integrations/document_stores/opensolr/client.py

@@ -13,7 +13,10 @@ from __future__ import annotations
 import re
 import json
 import time
-from typing import Any, Dict, List, Optional, Tuple
+import hashlib
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
+from urllib.parse import quote
 
 import httpx
 
@@ -600,6 +603,142 @@ def build_instruction(context: str, query: str) -> str:
     )
 
 
+# --------------------------------------------------------------------------- #
+# PDF pages and date ranges                                                     #
+# --------------------------------------------------------------------------- #
+# A PDF indexed from its file (Data Ingestion with rtf, the web crawler) is one Solr
+# document per page: page_i is the page number, uri_id_s the id of the PDF it belongs to.
+PAGE_FIELD = "page_i"
+PARENT_FIELD = "uri_id_s"
+
+#: Other matching pages returned under each PDF when the results are grouped.
+GROUP_EXPAND_ROWS = 4
+
+#: Most pages one read_document call returns.
+READ_MAX_PAGES = 500
+
+_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
+_META_FIELD_RE = re.compile(r"[^a-z0-9_]+")
+
+
+def date_range_filter(date_from: Optional[str] = None, date_to: Optional[str] = None) -> Optional[str]:
+    """Solr filter on ``creation_date`` for a period, both ends included (``YYYY-MM-DD``).
+
+    Either end may be left out. Returns ``None`` when neither is given.
+    """
+    import datetime
+
+    bounds = []
+    for name, value, clock in (("date_from", date_from, "T00:00:00Z"), ("date_to", date_to, "T23:59:59Z")):
+        if value is None or str(value).strip() == "":
+            bounds.append("*")
+            continue
+        text = str(value).strip()
+        if not _DATE_RE.match(text):
+            raise ValueError(f"{name} must be YYYY-MM-DD, got {value!r}")
+        try:
+            datetime.date.fromisoformat(text)
+        except ValueError:
+            raise ValueError(f"{name} is not a real date: {value!r}") from None
+        bounds.append(text + clock)
+    if bounds == ["*", "*"]:
+        return None
+    if "*" not in bounds and bounds[0] > bounds[1]:
+        raise ValueError(f"date_from ({date_from}) is after date_to ({date_to})")
+    return f"creation_date:[{bounds[0]} TO {bounds[1]}]"
+
+
+def add_filters(params: Dict[str, Any], *filters: Optional[str]) -> Dict[str, Any]:
+    """Append filter queries to ``params["fq"]``, keeping the ones already there."""
+    extra = [f for f in filters if f]
+    if not extra:
+        return params
+    existing = params.get("fq")
+    if existing is None:
+        params["fq"] = extra
+    elif isinstance(existing, list):
+        params["fq"] = existing + extra
+    else:
+        params["fq"] = [existing] + extra
+    return params
+
+
+def apply_page_grouping(params: Dict[str, Any]) -> Dict[str, Any]:
+    """One result per PDF (its best page), with its other matching pages in Solr's expanded section.
+
+    Documents that are not pages of a PDF stay separate results (``nullPolicy=expand``). The
+    collapse sort is passed by reference so {!hybrid} ranks every page before the grouping.
+    """
+    add_filters(params, "{!collapse field=" + PARENT_FIELD + " nullPolicy=expand sort=$qgsort}")
+    params["qgsort"] = "score desc," + PAGE_FIELD + " asc"
+    params["expand"] = "true"
+    params["expand.rows"] = GROUP_EXPAND_ROWS
+    params["expand.sort"] = "score desc," + PAGE_FIELD + " asc"
+    return params
+
+
+def page_url(uri: str, page: Optional[int]) -> str:
+    """The link that opens a PDF at one page."""
+    if uri and page and "#" not in uri:
+        return f"{uri}#page={int(page)}"
+    return uri
+
+
+def _first(value: Any) -> Any:
+    return value[0] if isinstance(value, list) and value else value
+
+
+def page_info(solr_doc: Dict[str, Any], expanded: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
+    """Where a hit lives: ``uri``; for a page of a PDF also ``page``, ``page_url``, ``document_id``
+    and, when the results were grouped, the other matching ``more_pages`` of that PDF."""
+    uri = str(_first(solr_doc.get("uri")) or "")
+    out: Dict[str, Any] = {"uri": uri}
+    try:
+        page = int(_first(solr_doc.get(PAGE_FIELD)) or 0)
+    except (TypeError, ValueError):
+        page = 0
+    if page < 1:
+        return out
+    parent = str(_first(solr_doc.get(PARENT_FIELD)) or "")
+    out.update(page=page, page_url=page_url(uri, page), document_id=parent)
+    group = (expanded or {}).get(parent) if parent else None
+    if isinstance(group, dict) and group.get("docs"):
+        more = []
+        for d in group["docs"]:
+            try:
+                n = int(_first(d.get(PAGE_FIELD)) or 0)
+            except (TypeError, ValueError):
+                continue
+            if n > 0:
+                more.append({"page": n, "page_url": page_url(str(_first(d.get("uri")) or uri), n)})
+        out["more_pages"] = more
+        out["matching_pages"] = 1 + int(group.get("numFound") or len(more))
+    return out
+
+
+def list_files(path: Union[str, Path], pattern: str = "*", recursive: bool = True) -> List[Path]:
+    """The files under ``path`` (or ``path`` itself when it is a file), hidden ones left out."""
+    root = Path(path).expanduser()
+    if root.is_file():
+        return [root]
+    if not root.is_dir():
+        raise FileNotFoundError(f"No such file or folder: {path}")
+    found = root.rglob(pattern) if recursive else root.glob(pattern)
+    return sorted(
+        f for f in found
+        if f.is_file() and not any(part.startswith(".") for part in f.relative_to(root).parts)
+    )
+
+
+def file_source(root: Union[str, Path], file: Union[str, Path]) -> str:
+    """A file's name in the index: its path from the folder that was sent, the folder's name first."""
+    root_p = Path(root).expanduser().resolve()
+    file_p = Path(file).expanduser().resolve()
+    if root_p.is_file() or root_p == file_p:
+        return file_p.name
+    return (Path(root_p.name) / file_p.relative_to(root_p)).as_posix()
+
+
 class OpensolrError(RuntimeError):
     """Raised when an Opensolr API call fails."""
 
@@ -619,6 +758,8 @@ class OpensolrClient:
         self.api_key = api_key
         self._http = httpx.Client(timeout=timeout, follow_redirects=True)
         self._core_info_cache: Dict[str, Dict[str, Any]] = {}
+        # indexes whose schema has no PDF page fields: searched and read without them
+        self._flat_indexes: set = set()
 
     # ------------------------------------------------------------------ #
     # low level                                                          #
@@ -641,14 +782,17 @@ class OpensolrClient:
         url = f"{base}/{method}"
         data = {**self._auth_params(), **params}
         resp = self._post_with_retry(url=url, data=data, label=method)
+        return self._answer(method, resp)
+
+    def _answer(self, label: str, resp: "httpx.Response") -> Any:
         if resp.status_code >= 500:
-            raise OpensolrError(f"{method}: HTTP {resp.status_code}: {resp.text[:200]}")
+            raise OpensolrError(f"{label}: HTTP {resp.status_code}: {resp.text[:200]}")
         try:
             body = resp.json()
         except json.JSONDecodeError as exc:
-            raise OpensolrError(f"{method}: non-JSON response: {resp.text[:200]}") from exc
+            raise OpensolrError(f"{label}: non-JSON response: {resp.text[:200]}") from exc
         if isinstance(body, dict) and body.get("status") is False:
-            raise OpensolrError(f"{method}: {body.get('msg', body)}")
+            raise OpensolrError(f"{label}: {body.get('msg', body)}")
         return body
 
     def _post_with_retry(self, *, url: str, label: str, **kwargs: Any) -> "httpx.Response":
@@ -1091,5 +1235,341 @@ class OpensolrClient:
         resp.raise_for_status()
         return resp.json()
 
+    # ------------------------------------------------------------------ #
+    # PDF pages, documents, pictures                                     #
+    # ------------------------------------------------------------------ #
+
+    def select_grouped(self, index: str, params: Dict[str, Any], group_pages: bool = True) -> Dict[str, Any]:
+        """``solr_select`` with the pages of each PDF grouped under its best page.
+
+        See :func:`apply_page_grouping`. An index whose schema has no page fields is searched
+        as it is, and remembered, so it is asked once.
+        """
+        if not group_pages or index in self._flat_indexes:
+            return self.solr_select(index, params)
+        grouped = dict(params)
+        if isinstance(grouped.get("fq"), list):
+            grouped["fq"] = list(grouped["fq"])
+        apply_page_grouping(grouped)
+        try:
+            return self.solr_select(index, grouped)
+        except httpx.HTTPStatusError as exc:
+            if exc.response.status_code == 400 and PARENT_FIELD in exc.response.text:
+                self._flat_indexes.add(index)
+                return self.solr_select(index, params)
+            raise
+
+    def read_document(
+        self,
+        index: str,
+        document: str,
+        page_from: int = 1,
+        page_to: Optional[int] = None,
+    ) -> Dict[str, Any]:
+        """The whole text of one indexed document.
+
+        ``document`` is its id, the id it was sent with (``meta_ext_id``) or its URL. A PDF
+        indexed page by page comes back as its ``pages`` in order, ``page_from`` to
+        ``page_to`` (at most :data:`READ_MAX_PAGES` per call), with ``pages_indexed``; any
+        other document as one ``text``.
+        """
+        key = str(document or "").strip()
+        if not key:
+            raise ValueError("document is required")
+        if key.startswith(("http://", "https://")):
+            key = hashlib.md5(key.rstrip("/").encode("utf-8")).hexdigest()
+        first = max(1, int(page_from or 1))
+        last = int(page_to) if page_to else first + READ_MAX_PAGES - 1
+        if last < first:
+            raise ValueError(f"page_to ({page_to}) is before page_from ({page_from})")
+        last = min(last, first + READ_MAX_PAGES - 1)
+        esc = key.replace("\\", "\\\\").replace('"', '\\"')
+        fl = "id,uri,title,description,text,content_type,creation_date,meta_ext_id"
+        docs: List[Dict[str, Any]] = []
+        if index not in self._flat_indexes:
+            try:
+                body = self.solr_select(index, {
+                    "q": f'id:"{esc}" OR {PARENT_FIELD}:"{esc}" OR meta_ext_id:"{esc}"',
+                    "fq": f"(*:* -{PAGE_FIELD}:[* TO *]) OR {PAGE_FIELD}:[{first} TO {last}]",
+                    "sort": f"{PAGE_FIELD} asc",
+                    "rows": last - first + 1,
+                    "fl": f"{fl},{PAGE_FIELD},{PARENT_FIELD}",
+                })
+                docs = body.get("response", {}).get("docs", [])
+            except httpx.HTTPStatusError as exc:
+                if exc.response.status_code != 400 or (PAGE_FIELD not in exc.response.text and PARENT_FIELD not in exc.response.text):
+                    raise
+                self._flat_indexes.add(index)
+        if index in self._flat_indexes:
+            body = self.solr_select(index, {"q": f'id:"{esc}" OR meta_ext_id:"{esc}"', "rows": 1, "fl": fl})
+            docs = body.get("response", {}).get("docs", [])
+        if not docs:
+            raise OpensolrError(f"read_document: nothing in {index} for {document!r}")
+
+        def _text(d: Dict[str, Any]) -> str:
+            t = d.get("text")
+            return "\n".join(str(x) for x in t) if isinstance(t, list) else str(t or "")
+
+        head = docs[0]
+        uri = str(_first(head.get("uri")) or "")
+        out: Dict[str, Any] = {
+            "id": str(_first(head.get(PARENT_FIELD)) or _first(head.get("id")) or ""),
+            "uri": uri,
+            "title": str(_first(head.get("title")) or ""),
+            "content_type": str(_first(head.get("content_type")) or ""),
+            "date": str(_first(head.get("creation_date")) or ""),
+        }
+        pages = []
+        for d in docs:
+            try:
+                n = int(_first(d.get(PAGE_FIELD)) or 0)
+            except (TypeError, ValueError):
+                n = 0
+            if n > 0:
+                pages.append({"page": n, "page_url": page_url(uri, n), "text": _text(d)})
+        if not pages:
+            out["text"] = _text(head)
+            return out
+        esc_parent = out["id"].replace("\\", "\\\\").replace('"', '\\"')
+        total = self.solr_select(index, {"q": f'{PARENT_FIELD}:"{esc_parent}"', "rows": 0})
+        out.update(
+            pages_indexed=int(total.get("response", {}).get("numFound") or len(pages)),
+            page_from=pages[0]["page"],
+            page_to=pages[-1]["page"],
+            pages=pages,
+        )
+        return out
+
+    def describe_image(self, index: str, image_b64: str) -> Dict[str, Any]:
+        """What a picture shows (``caption``), the text printed on it (``text``, read by OCR),
+        its ``labels`` and any barcodes / QR ``codes``."""
+        ans = self.image_to_text(index, image_b64, top_k=8)
+        labels = [l["label"] for l in (ans.get("labels") or []) if isinstance(l, dict) and l.get("label")]
+        codes: List[str] = []
+        for c in ans.get("codes") or []:
+            value = c.get("text") if isinstance(c, dict) else (c if isinstance(c, str) else None)
+            if value:
+                codes.append(value)
+        mode = ans.get("mode") or ""
+        ocr = str(ans.get("ocr_text") or "").strip() or (str(ans.get("text") or "").strip() if mode == "ocr" else "")
+        caption = str(ans.get("caption") or "").strip() or (str(ans.get("text") or "").strip() if mode != "ocr" else "")
+        return {"caption": caption, "text": ocr, "labels": labels, "codes": codes}
+
+    # ------------------------------------------------------------------ #
+    # files: read on Opensolr (text, OCR, a PDF page by page), indexed  #
+    # ------------------------------------------------------------------ #
+
+    #: Seconds between two looks at files that are still being read.
+    FILE_POLL_SECONDS = 3.0
+
+    #: Job ids one file_extract_status call accepts.
+    FILE_STATUS_MAX = 20
+
+    @staticmethod
+    def _file_hash(path: Path) -> Tuple[str, int]:
+        digest = hashlib.sha1()
+        size = 0
+        with open(path, "rb") as fh:
+            for block in iter(lambda: fh.read(4 * 1024 * 1024), b""):
+                digest.update(block)
+                size += len(block)
+        return digest.hexdigest(), size
+
+    def upload_file(
+        self,
+        index: str,
+        file: Union[str, Path],
+        pages: bool = False,
+        text: bool = True,
+    ) -> Dict[str, Any]:
+        """Send one local file to be read on Opensolr: its text, its pictures through OCR, a PDF page by page.
+
+        The bytes travel in parts, and only once: a file this account already sent (same
+        content) is answered without sending it again. Returns ``job_id`` and ``state``
+        (``queued``, ``processing``, ``done`` or ``failed``); once done, ``mime`` and the
+        ``text``, or with ``pages=True`` the ``pages`` of a PDF (``page``, ``text``,
+        ``is_ocr``, ``lang``). ``text=False`` answers the state alone. The pictures read are
+        counted on the index owner's AI quota, 0.5 each.
+        """
+        path = Path(file).expanduser()
+        sha1, size = self._file_hash(path)
+        if size < 1:
+            raise OpensolrError(f"file_extract: {path.name} is empty")
+        form = {
+            "index_name": index, "sha1": sha1, "size": size, "name": path.name,
+            "pages": "1" if pages else "0", "text": "1" if text else "0",
+        }
+        ans = self.ai("file_extract", **form)
+        stuck = 0
+        with open(path, "rb") as fh:
+            while ans.get("state") == "upload":
+                offset = int(ans.get("received") or 0)
+                part_max = int(ans.get("max_part_bytes") or 0)
+                if part_max < 1 or offset >= size:
+                    raise OpensolrError(f"file_extract: {path.name}: unexpected upload state {str(ans)[:200]}")
+                fh.seek(offset)
+                part = fh.read(min(part_max, size - offset))
+                resp = self._post_with_retry(
+                    label="file_extract",
+                    url=f"{AI_BASE}/file_extract",
+                    data={**self._auth_params(), **form, "offset": offset},
+                    files={"part": (path.name, part, "application/octet-stream")},
+                )
+                ans = self._answer("file_extract", resp)
+                if ans.get("state") == "upload" and int(ans.get("received") or 0) <= offset:
+                    stuck += 1
+                    if stuck >= 3:
+                        raise OpensolrError(f"file_extract: {path.name}: the upload does not get past {offset} bytes")
+                else:
+                    stuck = 0
+        return ans
+
+    def file_status(
+        self,
+        index: str,
+        job_ids: Sequence[str],
+        pages: bool = False,
+        text: bool = True,
+    ) -> Dict[str, Dict[str, Any]]:
+        """State of files sent with :meth:`upload_file`, keyed by job id, with their text (or pages) once read."""
+        ids = list(dict.fromkeys(str(j) for j in job_ids if j))
+        out: Dict[str, Dict[str, Any]] = {}
+        for i in range(0, len(ids), self.FILE_STATUS_MAX):
+            body = self.ai(
+                "file_extract_status", index_name=index, job_ids=json.dumps(ids[i : i + self.FILE_STATUS_MAX]),
+                pages="1" if pages else "0", text="1" if text else "0",
+            )
+            results = body.get("results") if isinstance(body, dict) else None
+            if not isinstance(results, dict):
+                raise OpensolrError(f"file_extract_status: unexpected response: {str(body)[:200]}")
+            out.update(results)
+        return out
+
+    def _wait_files(self, index: str, entries: List[Dict[str, Any]], pages: bool, text: bool, timeout: float) -> None:
+        deadline = time.time() + max(0.0, float(timeout))
+        while True:
+            waiting: Dict[str, List[Dict[str, Any]]] = {}
+            for e in entries:
+                if e.get("job_id") and e.get("state") in ("queued", "processing"):
+                    waiting.setdefault(e["job_id"], []).append(e)
+            if not waiting or time.time() >= deadline:
+                return
+            time.sleep(self.FILE_POLL_SECONDS)
+            for job_id, ans in self.file_status(index, list(waiting), pages=pages, text=text).items():
+                for e in waiting.get(job_id, []):
+                    e["state"] = ans.get("state") or "failed"
+                    for key in ("text", "pages", "mime", "error"):
+                        if key in ans:
+                            e[key] = ans[key]
+
+    def _send_files(self, index: str, files: Sequence[Union[str, Path]], pages: bool, text: bool) -> List[Dict[str, Any]]:
+        entries = []
+        for f in files:
+            entry: Dict[str, Any] = {"path": str(f)}
+            try:
+                ans = self.upload_file(index, f, pages=pages, text=text)
+                entry["job_id"] = ans.get("job_id")
+                entry["state"] = ans.get("state")
+                for key in ("text", "pages", "mime", "error"):
+                    if key in ans:
+                        entry[key] = ans[key]
+            except (OpensolrError, OSError) as exc:
+                entry.update(state="failed", error=str(exc))
+            entries.append(entry)
+        return entries
+
+    def extract_files(
+        self,
+        index: str,
+        files: Sequence[Union[str, Path]],
+        pages: bool = True,
+        timeout: float = 600.0,
+    ) -> List[Dict[str, Any]]:
+        """Read local files on Opensolr without indexing them (see :meth:`upload_file`).
+
+        One entry per file: ``path``, ``job_id``, ``state`` and, once ``done``, ``mime`` plus
+        the ``pages`` of a PDF (with ``pages=True``) or the ``text``; ``error`` when it could
+        not be read. Files still ``queued``/``processing`` when ``timeout`` runs out can be
+        asked for again: the bytes are not sent twice.
+        """
+        entries = self._send_files(index, files, pages=pages, text=True)
+        self._wait_files(index, entries, pages=pages, text=True, timeout=timeout)
+        return entries
+
+    def ingest_files(
+        self,
+        index: str,
+        path: Union[str, Path],
+        pattern: str = "*",
+        recursive: bool = True,
+        metadata: Optional[Dict[str, Any]] = None,
+        wait: bool = False,
+        timeout: float = 600.0,
+    ) -> Dict[str, Any]:
+        """Index local files through Data Ingestion: one file, or every file of a folder.
+
+        Each file is read on Opensolr (its text, its pictures through OCR) and a PDF is
+        indexed one document per page, exactly like a PDF sent by URL with ``rtf``. Its
+        ``uri`` is ``https://ingest.opensolr.com/<index>/<source>``, where ``source`` is the
+        file's path from the folder sent (also in ``meta_source``), so sending it again
+        replaces it; its date is the file's modification time. ``metadata`` is added to every
+        document as ``meta_*`` fields. Files still being read when ``timeout`` runs out come
+        back under ``pending``: call again to index them, the bytes are not sent twice.
+
+        Returns ``{"documents": [...], "pending": [...], "failed": [...], "jobs": [...]}``;
+        ``jobs`` are the Data Ingestion job ids (``wait=True`` blocks until they are done).
+        """
+        meta = {str(k): v for k, v in (metadata or {}).items() if isinstance(v, (str, int, float, bool))}
+        root = Path(path).expanduser()
+        files = list_files(root, pattern, recursive)
+        entries = self._send_files(index, files, pages=False, text=False)
+        for e, f in zip(entries, files):
+            e["source"] = file_source(root, f)
+        self._wait_files(index, entries, pages=False, text=False, timeout=timeout)
+
+        documents: List[Dict[str, Any]] = []
+        pending: List[Dict[str, Any]] = []
+        failed: List[Dict[str, Any]] = []
+        docs: List[Dict[str, Any]] = []
+        for e in entries:
+            if e.get("state") in ("queued", "processing"):
+                pending.append({"path": e["path"], "state": e["state"]})
+                continue
+            if e.get("state") != "done":
+                failed.append({"path": e["path"], "error": e.get("error") or e.get("state") or "not read"})
+                continue
+            src = e["source"]
+            uri = f"https://ingest.opensolr.com/{index}/{quote(src, safe='')}"
+            name = Path(e["path"])
+            doc: Dict[str, Any] = {
+                "uri": uri,
+                "title": (name.stem or name.name)[:250],
+                "description": src,
+                "rtf": True,
+                "file": e["job_id"],
+                "meta_ext_id": src,
+                "meta_source": src,
+                "meta_lc_json": json.dumps({"source": src, **meta}, ensure_ascii=False),
+            }
+            try:
+                doc["timestamp"] = int(name.stat().st_mtime)
+            except OSError:
+                pass
+            for key, value in meta.items():
+                field = "meta_" + _META_FIELD_RE.sub("_", key.lower()).strip("_")
+                if field not in ("meta_", "meta_lc_json", "meta_ext_id", "meta_source"):
+                    doc[field] = str(value)
+            docs.append(doc)
+            documents.append({
+                "path": e["path"], "source": src, "uri": uri,
+                "id": hashlib.md5(uri.encode("utf-8")).hexdigest(),
+            })
+        jobs: List[str] = []
+        for i in range(0, len(docs), 50):
+            body = self.ingest(index, docs[i : i + 50], wait=wait, timeout=timeout)
+            if isinstance(body, dict) and body.get("job_id"):
+                jobs.append(str(body["job_id"]))
+        return {"documents": documents, "pending": pending, "failed": failed, "jobs": jobs}
+
     def close(self) -> None:
         self._http.close()

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/haystack_integrations/document_stores/opensolr/store.py

@@ -11,7 +11,7 @@ import base64
 from pathlib import Path
 import json
 import re
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Union
 
 from haystack import Document, default_from_dict, default_to_dict
 from haystack.document_stores.types import DuplicatePolicy
@@ -22,6 +22,8 @@ from haystack_integrations.document_stores.opensolr.client import (
     OpensolrError,
     apply_fresh_bias,
     apply_search_operators,
+    date_range_filter,
+    page_info,
     parse_operators,
 )
 
@@ -198,7 +200,7 @@ class OpensolrDocumentStore:
                     continue
         self._checked = True
 
-    def _doc_from_solr(self, solr_doc: Dict[str, Any]) -> Document:
+    def _doc_from_solr(self, solr_doc: Dict[str, Any], expanded: Optional[Dict[str, Any]] = None) -> Document:
         def _flat(v: Any) -> Any:
             return v[0] if isinstance(v, list) and len(v) == 1 else v
 
@@ -214,8 +216,18 @@ class OpensolrDocumentStore:
             content = " ".join(str(c) for c in content)
         score = solr_doc.get("score")
         ext = _flat(solr_doc.get("meta_ext_id"))
+        doc_id = str(ext) if ext else str(_flat(solr_doc.get("id", "")))
+        # Where the hit lives: uri; a page of a PDF adds page_number, page_url, document_id, more_pages
+        where = page_info(solr_doc, expanded)
+        if where.get("page"):
+            where["page_number"] = where.pop("page")
+            # every page of a PDF carries the PDF's meta_ext_id: the page number keeps ids apart
+            doc_id = f"{doc_id}#page={where['page_number']}"
+        for key, value in where.items():
+            if value not in ("", None):
+                meta.setdefault(key, value)
         return Document(
-            id=str(ext) if ext else str(_flat(solr_doc.get("id", ""))),
+            id=doc_id,
             content=str(content),
             meta=meta,
             score=float(_flat(score)) if score is not None else None,
@@ -278,6 +290,34 @@ class OpensolrDocumentStore:
             self.client.ingest(self.index, docs[i : i + 50], wait=self.ingest_wait)
         return len(docs)
 
+    def write_files(
+        self,
+        path: Union[str, Path],
+        pattern: str = "*",
+        recursive: bool = True,
+        metadata: Optional[Dict[str, Any]] = None,
+        wait: Optional[bool] = None,
+        timeout: float = 600.0,
+    ) -> Dict[str, Any]:
+        """Index local files: one file, or every file of a folder.
+
+        Each file is read on Opensolr (its text, its pictures through OCR) and a PDF is
+        indexed one document per page, so a search finds the page that answers it.
+        Word, RTF, OpenDocument, text, HTML and picture files (scans) are one document
+        each. ``pattern`` picks a folder's files (``"*.pdf"``), ``metadata`` goes on
+        every document (filterable like any meta). Sending the same files again replaces
+        them. ``wait`` defaults to the store's ``ingest_wait``.
+
+        Returns ``{"documents", "pending", "failed", "jobs"}``: files still being read
+        when ``timeout`` ran out are under ``pending`` — call again to index them, the
+        bytes are not sent twice.
+        """
+        self._ensure_index()
+        return self.client.ingest_files(
+            self.index, path, pattern=pattern, recursive=recursive, metadata=metadata,
+            wait=self.ingest_wait if wait is None else wait, timeout=timeout,
+        )
+
     def ai_answer(
         self,
         query: str,
@@ -346,6 +386,9 @@ class OpensolrDocumentStore:
         filters: Optional[Dict[str, Any]] = None,
         lexical: bool = False,
         fresh_bias: bool = False,
+        date_from: Optional[str] = None,
+        date_to: Optional[str] = None,
+        group_pages: bool = True,
     ) -> List[Document]:
         """Retrieve the top_k documents for a query.
 
@@ -354,9 +397,17 @@ class OpensolrDocumentStore:
         filters — the hit count is unchanged, nothing old becomes unreachable, and
         a document with no ``creation_date`` is simply left unboosted. Applies to
         the lexical, hybrid and pure-kNN paths alike. Off by default.
+
+        ``date_from`` / ``date_to`` (``YYYY-MM-DD``, both included) keep only documents
+        dated in that period (``creation_date``). ``group_pages`` (on by default) returns
+        a PDF indexed page by page once, as its best page: ``meta`` carries
+        ``page_number``, ``page_url`` (opens the PDF at that page), ``document_id`` and
+        ``more_pages`` (its other matching pages). Every document carries ``uri``.
         """
         self._ensure_index()
         params: Dict[str, Any] = {"rows": top_k, "fl": "*,score"}
+        # Checked before anything is embedded: a malformed date is the caller's, not a GPU call's
+        period = date_range_filter(date_from, date_to)
         # Search operators (+word, -word, +"phrase", -"phrase") come out of the text once,
         # for every shape below. See parse_operators() for why they cannot stay inside the
         # query on any path that involves a vector.
@@ -378,9 +429,10 @@ class OpensolrDocumentStore:
             # top-level bf is not reliably its own.
             if fresh_bias:
                 apply_fresh_bias(params)
-            _extend_fq(params, _filters_to_fq(filters))
-            body = self.client.solr_select(self.index, params)
-            return [self._doc_from_solr(d) for d in body["response"]["docs"]]
+            _extend_fq(params, _filters_to_fq(filters) + ([period] if period else []))
+            body = self.client.select_grouped(self.index, params, group_pages)
+            expanded = body.get("expanded") or {}
+            return [self._doc_from_solr(d, expanded) for d in body["response"]["docs"]]
 
         # The embedder must never see an operator — it has no concept of negation, so a '-'
         # term reads as one more word and pulls results towards what the caller asked to drop.
@@ -407,10 +459,11 @@ class OpensolrDocumentStore:
         # including the vector-only ones an edismax bf never sees.
         if fresh_bias:
             apply_fresh_bias(params)
-        _extend_fq(params, _filters_to_fq(filters))
+        _extend_fq(params, _filters_to_fq(filters) + ([period] if period else []))
 
-        body = self.client.solr_select(self.index, params)
-        return [self._doc_from_solr(d) for d in body["response"]["docs"]]
+        body = self.client.select_grouped(self.index, params, group_pages)
+        expanded = body.get("expanded") or {}
+        return [self._doc_from_solr(d, expanded) for d in body["response"]["docs"]]
 
     # ------------------------------------------------------------------ #
     # serialization                                                      #
@@ -426,7 +479,7 @@ class OpensolrDocumentStore:
         """
         raw = bytes(image) if isinstance(image, (bytes, bytearray)) else Path(image).read_bytes()
         b64 = base64.b64encode(raw).decode("ascii")
-        ans = self.client.image_to_text(self._index, b64, top_k=top_k)
+        ans = self.client.image_to_text(self.index, b64, top_k=top_k)
         labels = [l["label"] for l in (ans.get("labels") or []) if isinstance(l, dict) and l.get("label")]
         codes = []
         for c in (ans.get("codes") or []):
@@ -447,8 +500,9 @@ class OpensolrDocumentStore:
         """Search the index with a photo instead of a text query.
 
         The image is turned into words by :meth:`image_to_words`, then those words
-        run through the normal :meth:`search`, so ``hybrid``, ``lexical``, ``alpha``
-        and ``fresh_bias`` apply via ``search_kwargs``. No image vector is stored.
+        run through the normal :meth:`search`, so ``hybrid``, ``lexical``, ``alpha``,
+        ``fresh_bias``, ``date_from``, ``date_to`` and ``group_pages`` apply via
+        ``search_kwargs``. No image vector is stored.
 
         ``using``: which reading to search with — ``"auto"`` (the engine's chosen
         text), ``"meaning"`` (CLIP visual labels), ``"text"`` (OCR text only), or

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/opensolr_haystack.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opensolr-haystack
-Version: 0.6.0
+Version: 0.7.0
 Summary: Haystack integration for Opensolr — managed Apache Solr DocumentStore with server-side embeddings and hybrid BM25+kNN retrieval
 Author-email: Opensolr <support@opensolr.com>
 License: MIT
@@ -61,8 +61,8 @@ the query. The store embeds server-side at both index and query time.
 
 ## Try it without an account
 
-There is a public demo account. Point the package at it and everything in this
-README works immediately, with no signup:
+There is a public demo account. Point the package at it and search, ingestion and
+answers work immediately, with no signup:
 
 ```bash
 export OPENSOLR_EMAIL=mcp@opensolr.com
@@ -72,6 +72,8 @@ export OPENSOLR_API_KEY=420b8b23e7b12dc8ab838932145a5065
 `mcp_demo_d1__dense` is already loaded with 300 news articles, so search, filtering
 and grounded answers work the moment you connect. You also get the full write path:
 create your own index on the account, ingest into it, query it, delete it.
+Reading files and pictures (the file loader and file indexing below, search by image)
+needs your own key: the shared demo key does not include them.
 
 Know what you are working with:
 
@@ -198,6 +200,63 @@ docs = store.search_by_image("photo.jpg", using="all")      # labels + OCR + cod
 `alpha`, `fresh_bias`, `filters` — so an image query runs through the exact
 hybrid pipeline a text query does.
 
+## Local files: PDFs page by page, scans through OCR
+
+`OpensolrConverter` turns files into Haystack `Document`s — one per PDF page
+(`meta["page_number"]`), one per any other file. It takes file paths, folders and
+`ByteStream`s:
+
+```python
+from haystack_integrations.components.converters.opensolr import OpensolrConverter
+
+converter = OpensolrConverter(document_store=store)
+docs = converter.run(sources=["./contracts", "scan.png"], meta={"team": "legal"})["documents"]
+# meta: file_path, page_number, is_ocr, language, content_type (+ yours)
+```
+
+Or index them straight into the store — no converter, no splitter, PDFs page by page:
+
+```python
+result = store.write_files("./contracts", pattern="*.pdf", metadata={"team": "legal"})
+# {"documents": [{path, source, uri, id}], "pending": [], "failed": [], "jobs": [...]}
+```
+
+- Each file is read on Opensolr's servers: its text, the text in its pictures (OCR,
+  more than 100 languages), a short description of the pictures without text. Word,
+  RTF, OpenDocument, text, HTML and picture files (scans) are one document each; a
+  **PDF is one document per page**.
+- The bytes travel once: a file the account already sent (same content) is not
+  uploaded again. Pictures read by OCR count 0.5 of an AI request each; a file read
+  before is free.
+- Files still being read when `timeout` runs out are under `result["pending"]` (the
+  converter logs them): run again to get them, nothing is sent twice.
+- An indexed file's `uri` is `https://ingest.opensolr.com/<index>/<source>`, where
+  `source` is its path from the folder you sent (`meta["source"]`); sending it again
+  replaces it. Its date is the file's modification time.
+
+## PDF pages in search results
+
+On an index where PDFs are stored page by page (files indexed as above, PDFs sent to
+Data Ingestion with `rtf`, PDFs found by the Web Crawler), a search returns each PDF
+**once**, as its best page. Its metadata carries `page_number` (the page number),
+`page_url` (opens the PDF at that page, `...pdf#page=14`), `document_id`,
+`more_pages` (its other matching pages, each with `page` and `page_url`) and
+`matching_pages`; every hit carries its `uri`. `group_pages=False` returns the pages
+as separate results.
+
+## Searching a period
+
+`date_from` and `date_to` (`YYYY-MM-DD`, both included, either one can be left out)
+keep only the documents dated in that period (`creation_date`):
+
+```python
+docs = store.search("rate decision", top_k=5, date_from="2026-01-01", date_to="2026-03-31")
+# or in a pipeline:
+OpensolrHybridRetriever(document_store=store, date_from="2026-01-01")
+```
+
+They combine with every other option — filters, `fresh_bias`, hybrid or lexical.
+
 ## Your index schema
 
 Documents follow the Opensolr document model (`title`, `description`, `text`,

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/opensolr_haystack.egg-info/SOURCES.txt

@@ -1,6 +1,8 @@
 LICENSE
 README.md
 pyproject.toml
+haystack_integrations/components/converters/opensolr/__init__.py
+haystack_integrations/components/converters/opensolr/converter.py
 haystack_integrations/components/retrievers/opensolr/__init__.py
 haystack_integrations/components/retrievers/opensolr/retriever.py
 haystack_integrations/document_stores/opensolr/__init__.py

{opensolr_haystack-0.6.0 → opensolr_haystack-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "opensolr-haystack"
-version = "0.6.0"
+version = "0.7.0"
 description = "Haystack integration for Opensolr — managed Apache Solr DocumentStore with server-side embeddings and hybrid BM25+kNN retrieval"
 readme = "README.md"
 license = { text = "MIT" }