raglite-toolkit 1.2.0__tar.gz → 1.2.2__tar.gz

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/.github/workflows/pr-verify.yml

@@ -34,3 +34,12 @@ jobs:
 
       - name: Run tests
         run: python -m pytest tests/ -q
+
+      - name: Check out TypeScript SDK for shared fixtures
+        uses: actions/checkout@v4
+        with:
+          repository: creatorpiyush/raglite
+          path: .ts-sdk
+
+      - name: Check shared fixtures match the TypeScript SDK
+        run: python scripts/sync_shared_fixtures.py --check --source .ts-sdk

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/.gitignore

@@ -42,3 +42,4 @@ coverage.xml
 
 # Logs
 *.log
+.ts-sdk/

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/ARCHITECTURE.md

@@ -126,9 +126,9 @@ graph TD
 ### 4.1 Ingestion & Indexing
 1. `doc.build()` invokes the appropriate `BaseLoader` based on file extension (`.pdf`, `.txt`, `.json`, `.md`, `.docx`).
 2. Calculates SHA-256 hash of raw document content.
-3. Checks existing `IndexMetadata` in `VectorStore`. If hash matches, skips re-indexing.
+3. Checks existing `IndexMetadata` in `VectorStore`. The cached index is reused when the index format version, content hash, chunk size, overlap and embedding provider/model all match (URL sources are hashed by their fetched text).
 4. If hash differs or force rebuild requested:
-   - `RecursiveCharacterTextSplitter` chunks text (default size 1000, overlap 200).
+   - `RecursiveChunker` splits text into word-based chunks (default 500 words, 50-word overlap).
    - `EmbeddingFactory` generates normalized vectors for each chunk.
    - `VectorStore.add()` saves chunks and `VectorStore.save_index_metadata()` persists index metadata.
 
@@ -162,27 +162,32 @@ Built using **FastAPI** framework for async capabilities, automatic OpenAPI docs
 ```python
 from abc import ABC, abstractmethod
 from typing import List, Optional
-from raglite.types import IndexMetadata, SearchResult, StoredChunk
+from raglite.types import IndexMetadata, StoredChunk
+from raglite.vectordb.base import VectorSearchHit
 
 class VectorStore(ABC):
+    @property
     @abstractmethod
-    def load(self, doc_id: str) -> None: ...
+    def namespace(self) -> str: ...
 
     @abstractmethod
-    def reset(self, doc_id: str) -> None: ...
+    def load(self) -> None: ...
 
     @abstractmethod
-    def add(self, doc_id: str, chunks: List[StoredChunk]) -> None: ...
+    def reset(self) -> None: ...
 
     @abstractmethod
-    def search(self, doc_id: str, query_vector: List[float], top_k: int, min_score: Optional[float] = None) -> List[SearchResult]: ...
+    def add(self, chunks: List[StoredChunk]) -> None: ...
 
     @abstractmethod
-    def count(self, doc_id: str) -> int: ...
+    def search(self, embedding: List[float], top_k: int) -> List[VectorSearchHit]: ...
 
     @abstractmethod
-    def save_index_metadata(self, doc_id: str, meta: IndexMetadata) -> None: ...
+    def count(self) -> int: ...
 
     @abstractmethod
-    def read_index_metadata(self, doc_id: str) -> Optional[IndexMetadata]: ...
+    def save_index_metadata(self, metadata: IndexMetadata) -> None: ...
+
+    @abstractmethod
+    def read_index_metadata(self) -> Optional[IndexMetadata]: ...
 ```

raglite_toolkit-1.2.2/CHANGELOG.md

@@ -0,0 +1,76 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [1.2.2] - Unreleased
+
+### Changed
+- **Upgrades keep cached indexes:** The build cache is now keyed on an index format version (`formatVersion` in `IndexMetadata`) instead of the package version, so upgrading RAGLite no longer re-embeds every index. Indexes built by 1.2.1 are reused as-is; indexes from older releases are rebuilt once.
+
+### Fixed
+- **Chunker parity with the TypeScript SDK:** Words are now split on exactly the whitespace characters JavaScript treats as whitespace. Previously text containing a byte order mark (U+FEFF), U+0085 or U+001C–U+001F was chunked differently from the TypeScript SDK. Existing indexes are not rebuilt for this; pass `rebuild=True` if your sources contain those characters.
+- **Docs:** ARCHITECTURE.md now describes the word-based `RecursiveChunker` (500 words, 50 overlap) and the actual `VectorStore` interface. The README no longer lists LanceDB, which is TypeScript-only.
+
+### Internal
+- Added cross-SDK fixtures (`tests/fixtures/shared/`, copied from the TypeScript SDK with `scripts/sync_shared_fixtures.py`) that pin chunking and hashing output. CI fails if the copy is out of date.
+
+## [1.2.1] - 2026-10-02
+
+### Fixed
+- **URL sources:** `Document.build()` no longer fails with `LoaderError: File does not exist` for web URLs, so URLs work on their own and inside a `DocumentCollection`.
+- **Custom `VectorStore` instances:** Passing a `VectorStore` subclass instance as `{"vectorStore": store}` no longer fails config validation, so the documented custom store usage works.
+- **Qdrant shared collections:** When `indexName` is set, documents sharing one Qdrant collection no longer wipe each other. `reset()`, search and index metadata are now scoped to each document's namespace instead of the whole collection. Without `indexName`, behaviour is unchanged.
+- **Shared `VectorStore` instances in collections:** `DocumentCollection.build()` now raises `ConfigError` when a single `VectorStore` instance would be shared by more than one document, instead of each document silently resetting the previous one's index. Pass a vector store provider config instead.
+- **Web URL re-indexing:** URL sources are now fingerprinted by their fetched content rather than the URL string, so a changed page is re-indexed on the next `build()`.
+- **Query embedder after reload:** Searching an existing index in a new process now embeds queries with the provider and model the index was built with, rather than the constructor default. Configured credentials are reused when the provider matches.
+- **Collection search errors:** `DocumentCollection.search()` (and `ask`/`ask_stream`) now logs per-document search failures instead of silently dropping them, and raises if every document fails.
+- **Web loader User-Agent:** Now reports the actual package version.
+
+### Upgrade notes
+- As with every release, cached indexes are rebuilt once on first `build()` because the package version is part of the cache key.
+- Existing Qdrant indexes created with `indexName` are rebuilt once. Their old untagged points stay in the collection but are no longer returned by search; drop and recreate the collection to remove them.
+- Each `build()` on a URL source now fetches the page to check for changes, even when the cached index is reused.
+- When every document in a collection fails to search, the REST server now returns an error response (400 for RAGLite errors, 500 otherwise) instead of empty results.
+
+## [1.2.0] - 2026-08-02
+
+### Added
+- **Multi-Document & Directory Ingestion (`DocumentCollection`):**
+  - Added `DocumentCollection` class to manage semantic indexing, multi-document retrieval, and Q&A across folders, glob patterns, web URLs, and mixed file lists.
+  - Parallel semantic search over collection vector stores with score-based top-$K$ merging and ranking.
+  - Contextual Q&A synthesis (`ask` and `ask_stream`) across multi-document collections.
+- **Directory Loader (`DirectoryLoader`):**
+  - Recursive directory scanner (`recursive=True`) with glob pattern matching (e.g. `./docs/**/*.md`).
+  - Auto-detection of supported extensions (`.pdf`, `.txt`, `.md`, `.json`, `.docx`).
+  - Detailed error reporting and warning logs for unsupported/empty files.
+- **Web Loader (`WebLoader`):**
+  - Native loader for fetching HTTP/HTTPS web URLs directly.
+  - Automatic HTML cleaning into formatted text/markdown with script, style, and SVG tag stripping.
+  - JSON and plain text content-type parsing.
+- **CLI & FastAPI REST Server Support:**
+  - Upgraded `raglite index`, `search`, `ask`, and `serve` CLI commands to process directories, glob patterns, and URLs.
+  - Updated FastAPI REST server to support `DocumentCollection` and single `Document` targets.
+- **Unit & Integration Tests:**
+  - Added unit and integration tests for `DirectoryLoader`, `WebLoader`, and `DocumentCollection`.
+
+## [1.1.0] - 2026-07-19
+
+### Added
+- **Pluggable Vector Databases:** Added support for custom local and cloud vector database backends via a new `vectorStore` config option.
+  - **Memory Store (`"memory"`):** Default in-memory store persisting indexes locally to JSON (unchanged behaviour).
+  - **Qdrant Store (`"qdrant"`):** Wrapper for Qdrant local/cloud using stdlib `urllib` REST requests. Supports auto-collection creation, API key auth, and custom collection names.
+  - **Pinecone Store (`"pinecone"`):** Cloud database support using Pinecone Namespaces and stdlib `urllib` REST requests. Stores index metadata as a reserved `__metadata__` vector.
+  - **Custom Adapters:** Pass any class instance implementing the `VectorStore` ABC directly as `vectorStore` in `DocumentOptions`.
+- **`VectorStoreProviderConfig` type:** New Pydantic model in `types.py` describing provider, URL, API key, index name, and store directory.
+- **Factory:** `create_vector_store(config, namespace)` utility in `vectordb/factory.py` resolving the correct store from config.
+- **Examples:**
+  - `examples/qdrant_example.py` — full index + search demo with Qdrant.
+  - `examples/custom_store_example.py` — implementing and using a custom VectorStore ABC subclass.
+- **Automation Scripts:**
+  - `scripts/pre-commit.sh` — runs ruff lint, mypy type-check, and pytest before committing.
+  - `scripts/pre-release.sh` — cleans builds, runs full verification, and builds distribution packages.
+- **GitHub Actions Workflow:** `pr-verify.yml` — automatically runs code style checks and the full test suite on every pull request and push to `main`/`master`.
+
+### Changed
+- **`DocumentOptions` / `ResolvedConfig`:** Added optional `vectorStore` field supporting `VectorStoreProviderConfig` or a custom `VectorStore` instance.
+- **`Document.__init__`:** Constructor now resolves the appropriate vector store from config, accepting provider configs or custom instances.

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: raglite-toolkit
-Version: 1.2.0
+Version: 1.2.2
 Summary: Build semantic search, multi-provider question answering, and REST APIs over your documents in a few lines of Python.
 Project-URL: Homepage, https://github.com/creatorpiyush/raglite-py
 Project-URL: Repository, https://github.com/creatorpiyush/raglite-py
@@ -172,7 +172,7 @@ print()
 
 ## Pluggable Vector Databases
 
-`raglite` supports pluggable vector stores (Memory, Qdrant, Pinecone, LanceDB, or custom subclasses):
+`raglite` supports pluggable vector stores (Memory, Qdrant, Pinecone, or custom subclasses; LanceDB is currently TypeScript-only):
 
 ### Memory Store (Default)
 ```python
@@ -182,6 +182,7 @@ doc = Document("./policy.pdf", {
 ```
 
 ### Qdrant Store
+By default each document gets its own collection (`raglite_<namespace>`). Set `indexName` to keep several documents in one shared collection; each document's points are tagged with its namespace, so rebuilding one document never affects the others.
 ```python
 doc = Document("./policy.pdf", {
     "vectorStore": {
@@ -335,7 +336,7 @@ Document("./policy.pdf", {
 
 ## How Caching Works
 
-Every `build()` call fingerprints the source file with a **SHA-256 content hash** and persists it alongside the vectors. The cached index is reused only if **all** of the following match the stored index:
+Every `build()` call fingerprints the source (file bytes, or the fetched text for URLs) with a **SHA-256 content hash** and persists it alongside the vectors. The cached index is reused only if **all** of the following match the stored index:
 
 | Factor | Triggers rebuild if changed |
 |--------|-----------------------------|
@@ -343,7 +344,7 @@ Every `build()` call fingerprints the source file with a **SHA-256 content hash*
 | Chunk size | `chunkSize` changed |
 | Overlap | `overlap` changed |
 | Embedding provider/model | Provider or model string changed |
-| Library version | Package version bumped |
+| Index format | Stored index layout changed by a release (rare; ordinary upgrades reuse the index) |
 
 Pass `rebuild=True` to `build()` to force a fresh index regardless.
 
@@ -359,7 +360,7 @@ Each document is stored under `.raglite/<sha256-prefix>/`, so multiple documents
 from raglite.vectordb.base import VectorStore
 
 class MyVectorStore(VectorStore):
-    # Implement: load, reset, add, search, count,
+    # Implement: namespace (property), load, reset, add, search, count,
     #            save_index_metadata, read_index_metadata
     ...
 ```

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/README.md

@@ -126,7 +126,7 @@ print()
 
 ## Pluggable Vector Databases
 
-`raglite` supports pluggable vector stores (Memory, Qdrant, Pinecone, LanceDB, or custom subclasses):
+`raglite` supports pluggable vector stores (Memory, Qdrant, Pinecone, or custom subclasses; LanceDB is currently TypeScript-only):
 
 ### Memory Store (Default)
 ```python
@@ -136,6 +136,7 @@ doc = Document("./policy.pdf", {
 ```
 
 ### Qdrant Store
+By default each document gets its own collection (`raglite_<namespace>`). Set `indexName` to keep several documents in one shared collection; each document's points are tagged with its namespace, so rebuilding one document never affects the others.
 ```python
 doc = Document("./policy.pdf", {
     "vectorStore": {
@@ -289,7 +290,7 @@ Document("./policy.pdf", {
 
 ## How Caching Works
 
-Every `build()` call fingerprints the source file with a **SHA-256 content hash** and persists it alongside the vectors. The cached index is reused only if **all** of the following match the stored index:
+Every `build()` call fingerprints the source (file bytes, or the fetched text for URLs) with a **SHA-256 content hash** and persists it alongside the vectors. The cached index is reused only if **all** of the following match the stored index:
 
 | Factor | Triggers rebuild if changed |
 |--------|-----------------------------|
@@ -297,7 +298,7 @@ Every `build()` call fingerprints the source file with a **SHA-256 content hash*
 | Chunk size | `chunkSize` changed |
 | Overlap | `overlap` changed |
 | Embedding provider/model | Provider or model string changed |
-| Library version | Package version bumped |
+| Index format | Stored index layout changed by a release (rare; ordinary upgrades reuse the index) |
 
 Pass `rebuild=True` to `build()` to force a fresh index regardless.
 
@@ -313,7 +314,7 @@ Each document is stored under `.raglite/<sha256-prefix>/`, so multiple documents
 from raglite.vectordb.base import VectorStore
 
 class MyVectorStore(VectorStore):
-    # Implement: load, reset, add, search, count,
+    # Implement: namespace (property), load, reset, add, search, count,
     #            save_index_metadata, read_index_metadata
     ...
 ```

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/pyproject.toml

@@ -8,7 +8,7 @@ packages = ["src/raglite"]
 
 [project]
 name = "raglite-toolkit"
-version = "1.2.0"
+version = "1.2.2"
 description = "Build semantic search, multi-provider question answering, and REST APIs over your documents in a few lines of Python."
 readme = "README.md"
 requires-python = ">=3.10"

raglite_toolkit-1.2.2/scripts/sync_shared_fixtures.py

@@ -0,0 +1,54 @@
+"""Copy or check the cross-SDK fixtures shared with the TypeScript SDK.
+
+The TypeScript repo (raglite) is the source of truth: it generates
+tests/fixtures/shared/*.json with scripts/generate-shared-fixtures.ts. This
+repo keeps an identical copy so both SDKs are tested against the same
+expected outputs.
+
+    python scripts/sync_shared_fixtures.py            # copy from ../raglite
+    python scripts/sync_shared_fixtures.py --check    # exit 1 if out of date
+    python scripts/sync_shared_fixtures.py --source /path/to/raglite
+"""
+import argparse
+import filecmp
+import shutil
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+TARGET = REPO_ROOT / "tests" / "fixtures" / "shared"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--source", type=Path, default=REPO_ROOT.parent / "raglite")
+    parser.add_argument("--check", action="store_true")
+    args = parser.parse_args()
+
+    source = args.source / "tests" / "fixtures" / "shared"
+    files = sorted(source.glob("*.json"))
+    if not files:
+        print(f"No shared fixtures found in {source}", file=sys.stderr)
+        return 1
+
+    stale = [f.name for f in files if not (TARGET / f.name).exists() or not filecmp.cmp(f, TARGET / f.name, shallow=False)]
+    extra = sorted({p.name for p in TARGET.glob("*.json")} - {f.name for f in files})
+
+    if args.check:
+        for name in stale:
+            print(f"out of date: {name}", file=sys.stderr)
+        for name in extra:
+            print(f"not in source: {name}", file=sys.stderr)
+        return 1 if stale or extra else 0
+
+    TARGET.mkdir(parents=True, exist_ok=True)
+    for f in files:
+        shutil.copyfile(f, TARGET / f.name)
+    for name in extra:
+        (TARGET / name).unlink()
+    print(f"Synced {len(files)} fixture(s) from {source}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/chunking/recursive.py

@@ -4,17 +4,24 @@ from typing import List
 from ..errors import ChunkingError
 from .base import BaseChunker
 
+# The exact set JavaScript's \s matches. Python's \s differs (it includes
+# \x1c-\x1f and \x85 but not \ufeff), which would make the two SDKs chunk the
+# same text differently.
+_JS_WHITESPACE = re.compile(
+    "[\t\n\v\f\r \u00a0\u1680\u2000-\u200a\u2028\u2029\u202f\u205f\u3000\ufeff]+"
+)
+
 
 class RecursiveChunker(BaseChunker):
     def split(self, text: str) -> List[str]:
-        if not text.strip():
+        if not _JS_WHITESPACE.sub("", text):
             return []
         if self.overlap >= self.chunk_size:
             raise ChunkingError(
                 f"overlap ({self.overlap}) must be smaller than chunkSize ({self.chunk_size})"
             )
 
-        words = [w for w in re.split(r"\s+", text) if w]
+        words = [w for w in _JS_WHITESPACE.split(text) if w]
         if len(words) <= self.chunk_size:
             return [" ".join(words)]
 

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/config.py

@@ -10,6 +10,7 @@ from .constants import (
     DEFAULT_TOP_K,
 )
 from .types import EmbeddingProviderConfig, LLMProviderConfig, VectorStoreProviderConfig
+from .vectordb.base import VectorStore
 
 
 class DocumentOptions(BaseModel):
@@ -44,6 +45,10 @@ def resolve_config(options: Optional[Union[DocumentOptions, Dict[str, Any]]] = N
     if options is None:
         opts = DocumentOptions()
     elif isinstance(options, dict):
+        # A VectorStore instance is read straight from the raw options by
+        # Document, so keep it out of the provider-config validation.
+        if isinstance(options.get("vectorStore"), VectorStore):
+            options = {k: v for k, v in options.items() if k != "vectorStore"}
         opts = DocumentOptions.model_validate(options)
     else:
         opts = options

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/constants.py

@@ -5,7 +5,16 @@ try:
     PACKAGE_VERSION = importlib.metadata.version("raglite-toolkit")
 except importlib.metadata.PackageNotFoundError:
     PACKAGE_NAME = "raglite-toolkit"
-    PACKAGE_VERSION = "1.2.0"  # local development fallback
+    PACKAGE_VERSION = "1.2.2"  # local development fallback
+
+# Version of the stored index layout (chunking, ids, payloads). The build cache
+# is keyed on this instead of PACKAGE_VERSION so upgrading the package does not
+# re-embed every index. Bump it only when a change makes existing indexes
+# incompatible.
+INDEX_FORMAT_VERSION = 1
+
+# Releases that wrote format-1 indexes before ``formatVersion`` was recorded.
+LEGACY_FORMAT_1_VERSIONS = frozenset({"1.2.1"})
 
 
 SUPPORTED_EXTENSIONS = {".pdf", ".txt", ".json", ".md", ".markdown", ".docx"}

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/core/collection.py

@@ -4,11 +4,12 @@ from pathlib import Path
 from typing import Any, Dict, Generator, List, Optional, Union
 
 from ..config import DocumentOptions, ResolvedConfig, resolve_config
-from ..errors import RagLiteError
+from ..errors import ConfigError, RagLiteError
 from ..llm import generate_answer, stream_answer
 from ..loaders import DirectoryLoader, is_supported_file, is_url
 from ..types import AnswerResult, SearchResult
 from ..utils.logger import Logger, create_logger
+from ..vectordb import VectorStore
 from .document import Document
 
 
@@ -91,6 +92,16 @@ class DocumentCollection:
                     self.logger.warning(msg)
                     errors.append({"source": src, "error": msg})
 
+        # A VectorStore instance has a single namespace, so every document would
+        # share it and each build() would reset the previous document's index.
+        raw_store = self.options.get("vectorStore") if isinstance(self.options, dict) else None
+        if isinstance(raw_store, VectorStore) and len(set(file_list)) > 1:
+            raise ConfigError(
+                "A VectorStore instance cannot be shared by multiple documents in a "
+                "DocumentCollection. Pass a vector store provider config "
+                '(e.g. {"provider": "qdrant", ...}) instead.'
+            )
+
         total_chunks = 0
         cached_docs = 0
         new_docs = 0
@@ -146,12 +157,19 @@ class DocumentCollection:
         st = score_threshold if score_threshold is not None else opts.get("scoreThreshold", opts.get("score_threshold", self.config.scoreThreshold))
 
         all_hits: List[SearchResult] = []
+        failures: List[Exception] = []
         for doc in self.documents.values():
             try:
                 hits = doc.search(query, top_k=tk * 2, score_threshold=st)
                 all_hits.extend(hits)
-            except Exception:
-                continue
+            except Exception as err:
+                failures.append(err)
+                self.logger.warning(f'Search failed for document "{doc.file_path}": {err}')
+
+        # One broken document should not hide results from the others, but if
+        # every document failed the caller needs the error, not an empty list.
+        if len(failures) == len(self.documents):
+            raise failures[0]
 
         all_hits.sort(key=lambda h: h.score, reverse=True)
         return all_hits[:tk]

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/core/document.py

@@ -4,13 +4,19 @@ from typing import Any, Dict, Generator, Optional, Union
 
 from ..chunking import RecursiveChunker
 from ..config import DocumentOptions, ResolvedConfig, resolve_config
-from ..constants import PACKAGE_VERSION
+from ..constants import INDEX_FORMAT_VERSION, LEGACY_FORMAT_1_VERSIONS, PACKAGE_VERSION
 from ..embeddings import Embedder, create_embedder
 from ..errors import FileNotIndexedError, LoaderError, RagLiteError
 from ..llm import generate_answer, stream_answer
 from ..loaders import get_loader, is_url
 from ..retrieval import Retriever
-from ..types import AnswerResult, ChunkMetadata, IndexMetadata, StoredChunk
+from ..types import (
+    AnswerResult,
+    ChunkMetadata,
+    EmbeddingProviderConfig,
+    IndexMetadata,
+    StoredChunk,
+)
 from ..utils.hash import hash_file, hash_string, namespace_from_path
 from ..utils.logger import create_logger
 from ..vectordb import MemoryVectorStore, VectorStore, create_vector_store
@@ -74,16 +80,17 @@ class Document:
         if should_rebuild is None:
             should_rebuild = opts.get("rebuild", False)
 
-        if not os.path.exists(self.file_path):
+        if not is_url(self.file_path) and not os.path.exists(self.file_path):
             raise LoaderError(f"File does not exist: {self.file_path}")
 
         self.store.load()
         existing = self.store.read_index_metadata()
-        source_hash = (
-            hash_string(self.file_path)
-            if is_url(self.file_path)
-            else hash_file(self.file_path)
-        )
+        # Web pages change without their URL changing, so fingerprint the
+        # fetched content rather than the URL.
+        text: Optional[str] = None
+        if is_url(self.file_path):
+            text = get_loader(self.file_path).load()
+        source_hash = hash_string(text) if text is not None else hash_file(self.file_path)
 
         if (
             not should_rebuild
@@ -110,8 +117,8 @@ class Document:
         self.store.reset()
         self.store.load()
 
-        loader = get_loader(self.file_path)
-        text = loader.load()
+        if text is None:
+            text = get_loader(self.file_path).load()
         if not text:
             raise LoaderError(
                 f"Loader returned empty text for {self.file_path}"
@@ -161,6 +168,7 @@ class Document:
         from datetime import timezone
         metadata = IndexMetadata(
             version=PACKAGE_VERSION,
+            formatVersion=INDEX_FORMAT_VERSION,
             source=self.file_path,
             sourceHash=source_hash,
             chunkSize=c_size,
@@ -340,19 +348,10 @@ class Document:
                 f'No RagLite index found for "{self.file_path}". Call build() first.'
             )
 
-        embed_config = self.config.embeddings
-        if embed_config.model is None:
-            from ..types import EmbeddingProviderConfig
-
-            embed_config = EmbeddingProviderConfig(
-                provider=embed_config.provider,
-                model=existing.embeddingModel,
-                apiKey=embed_config.apiKey,
-                baseURL=embed_config.baseURL,
-            )
-
         if self.embedder is None:
-            self.embedder = create_embedder(embed_config)
+            self.embedder = create_embedder(
+                _query_embeddings_config(existing, self.config.embeddings)
+            )
         self.ready = True
 
     def _cache_still_valid(
@@ -363,7 +362,7 @@ class Document:
         overlap: int,
         embeddings_config: Any,
     ) -> bool:
-        if existing.version != PACKAGE_VERSION:
+        if index_format_version(existing) != INDEX_FORMAT_VERSION:
             return False
         if existing.sourceHash != source_hash:
             return False
@@ -378,3 +377,26 @@ class Document:
         if req_model is not None and req_model != existing.embeddingModel:
             return False
         return True
+
+
+def index_format_version(metadata: IndexMetadata) -> Optional[int]:
+    """Indexes written before ``formatVersion`` existed are identified by package version."""
+    if metadata.formatVersion is not None:
+        return metadata.formatVersion
+    return 1 if metadata.version in LEGACY_FORMAT_1_VERSIONS else None
+
+
+def _query_embeddings_config(
+    existing: IndexMetadata, configured: EmbeddingProviderConfig
+) -> EmbeddingProviderConfig:
+    """Queries must be embedded with the same provider and model as the index,
+    which may differ from the constructor default when ``build(embeddings=...)``
+    overrode it. Credentials from the configured provider are reused when it
+    matches; otherwise the provider falls back to its environment variables."""
+    same_provider = configured.provider == existing.embeddingProvider
+    return EmbeddingProviderConfig(
+        provider=existing.embeddingProvider,
+        model=existing.embeddingModel,
+        apiKey=configured.apiKey if same_provider else None,
+        baseURL=configured.baseURL if same_provider else None,
+    )

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/loaders/web.py

@@ -1,6 +1,7 @@
 import html
 import re
 
+from ..constants import PACKAGE_VERSION
 from ..errors import LoaderError
 from .base import BaseLoader
 
@@ -18,7 +19,7 @@ class WebLoader(BaseLoader):
             import httpx
 
             headers = {
-                "User-Agent": "RAGLite/1.2.0 (Python/3.10+)",
+                "User-Agent": f"RAGLite/{PACKAGE_VERSION} (Python/3.10+)",
                 "Accept": "text/html,text/plain,application/xhtml+xml;q=0.9,*/*;q=0.8",
             }
             response = httpx.get(self.url, headers=headers, follow_redirects=True, timeout=15.0)

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/types.py

@@ -96,7 +96,9 @@ class AnswerResult(BaseModel):
 class IndexMetadata(BaseModel):
     model_config = ConfigDict(populate_by_name=True, extra="allow")
 
-    version: str
+    version: str  # package version that built the index (informational)
+    # Index layout version; see INDEX_FORMAT_VERSION. None on indexes built before 1.2.2.
+    formatVersion: Optional[int] = Field(default=None, alias="formatVersion")
     source: str
     sourceHash: str = Field(..., alias="sourceHash")
     chunkSize: int = Field(..., alias="chunkSize")

{raglite_toolkit-1.2.0 → raglite_toolkit-1.2.2}/src/raglite/vectordb/pinecone.py

@@ -126,7 +126,8 @@ class PineconeVectorStore(VectorStore):
     def save_index_metadata(self, metadata: IndexMetadata) -> None:
         dim = metadata.embeddingDimensions
         zero_vec = [0.0] * dim
-        payload = metadata.model_dump(by_alias=True)
+        # Pinecone rejects null metadata values.
+        payload = metadata.model_dump(by_alias=True, exclude_none=True)
         payload["isMetadata"] = True
         self._request(
             "POST",
@@ -154,6 +155,9 @@ class PineconeVectorStore(VectorStore):
         try:
             return IndexMetadata(
                 version=m.get("version", ""),
+                formatVersion=(
+                    int(m["formatVersion"]) if m.get("formatVersion") is not None else None
+                ),
                 source=m.get("source", ""),
                 sourceHash=m.get("sourceHash", ""),
                 chunkSize=int(m.get("chunkSize", 0)),