bilbysearch 0.1.0__tar.gz

bilbysearch-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+# Python / uv
+.venv/
+__pycache__/
+*.py[cod]
+.ruff_cache/
+
+# Build artifacts
+dist/
+*.egg-info/
+
+# OS
+.DS_Store
+
+# Claude files
+CLAUDE.md

bilbysearch-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

bilbysearch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Samson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bilbysearch-0.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.5
+Name: bilbysearch
+Version: 0.1.0
+Summary: Article embeddings with intfloat/multilingual-e5-large-instruct.
+Project-URL: Homepage, https://github.com/bilbyai/bilbysearch
+Project-URL: Repository, https://github.com/bilbyai/bilbysearch
+Author: Samson
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: pyarrow
+Provides-Extra: all
+Requires-Dist: fsspec; extra == 'all'
+Requires-Dist: gcsfs; extra == 'all'
+Requires-Dist: google-cloud-storage; extra == 'all'
+Requires-Dist: sentence-transformers; extra == 'all'
+Requires-Dist: torch; extra == 'all'
+Provides-Extra: embed
+Requires-Dist: sentence-transformers; extra == 'embed'
+Requires-Dist: torch; extra == 'embed'
+Provides-Extra: gcs
+Requires-Dist: fsspec; extra == 'gcs'
+Requires-Dist: gcsfs; extra == 'gcs'
+Requires-Dist: google-cloud-storage; extra == 'gcs'
+Description-Content-Type: text/markdown
+
+# bilbysearch
+
+**bilbysearch** turns articles into vectors with
+[intfloat/multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct),
+and reads and writes the daily parquet files those vectors live in.
+
+It exists so the model's contract lives in one place. The model id, the 1024-d output
+width, the instruction prefix queries need and the `title+body` join rule are all in
+[`config.py`](bilbysearch/config.py), so notebooks, batch backfills and retrieval
+pipelines cannot drift apart.
+
+## Install
+
+```bash
+pip install bilbysearch[embed]
+```
+
+The base install is deliberately light — just pandas and pyarrow — so a caller that only
+reads the parquet files, or only reads the model contract, does not pull a multi-gigabyte
+ML stack it will never use.
+
+| Extra | Adds | Needed for |
+|---|---|---|
+| *(none)* | pandas, pyarrow, numpy | reading `config`, local parquet IO |
+| `[embed]` | sentence-transformers, torch | `load_embedder` |
+| `[gcs]` | fsspec, gcsfs, google-cloud-storage | `gs://` URIs |
+| `[all]` | both of the above | everything |
+
+Until the first PyPI release, install from the repo instead:
+
+```bash
+uv add "bilbysearch[all] @ git+https://github.com/bilbyai/bilbysearch.git"
+```
+
+## Usage
+
+```python
+import bilbysearch
+
+model = bilbysearch.load_embedder()
+
+# Embed a frame of articles. Rows with no text get a null vector rather than a
+# meaningless one, and three provenance columns record how the vectors were made.
+articles = bilbysearch.read_parquet("daily_2026-01-31.parquet")
+embedded = bilbysearch.embed_dataframe(articles, model=model)
+bilbysearch.write_parquet(embedded, "daily_2026-01-31_embedding.parquet")
+```
+
+Queries and passages are embedded by **different functions on purpose**: this model
+expects an instruction prefix on queries and none on passages. Skipping it measurably
+degrades retrieval.
+
+```python
+query = bilbysearch.embed_query("Chinese monetary policy easing", model=model)
+passages = bilbysearch.embed_texts(["The PBoC cut the reserve requirement ratio."], model=model)
+similarity = query @ passages[0]  # vectors are L2-normalised, so this is cosine
+```
+
+Only `load_embedder` needs the `[embed]` extra. Every other function works on any object
+exposing sentence-transformers' `encode`, so a caller that already has a model can pass
+it straight in.
+
+## Development
+
+```bash
+uv sync
+uv run pytest
+uv run ruff check . && uv run ruff format .
+```
+
+The tests use a stub model, so the suite runs in well under a second and never downloads
+the real 2.2 GB one.
+
+## Licence
+
+MIT — see [LICENSE](LICENSE).

bilbysearch-0.1.0/README.md

@@ -0,0 +1,76 @@
+# bilbysearch
+
+**bilbysearch** turns articles into vectors with
+[intfloat/multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct),
+and reads and writes the daily parquet files those vectors live in.
+
+It exists so the model's contract lives in one place. The model id, the 1024-d output
+width, the instruction prefix queries need and the `title+body` join rule are all in
+[`config.py`](bilbysearch/config.py), so notebooks, batch backfills and retrieval
+pipelines cannot drift apart.
+
+## Install
+
+```bash
+pip install bilbysearch[embed]
+```
+
+The base install is deliberately light — just pandas and pyarrow — so a caller that only
+reads the parquet files, or only reads the model contract, does not pull a multi-gigabyte
+ML stack it will never use.
+
+| Extra | Adds | Needed for |
+|---|---|---|
+| *(none)* | pandas, pyarrow, numpy | reading `config`, local parquet IO |
+| `[embed]` | sentence-transformers, torch | `load_embedder` |
+| `[gcs]` | fsspec, gcsfs, google-cloud-storage | `gs://` URIs |
+| `[all]` | both of the above | everything |
+
+Until the first PyPI release, install from the repo instead:
+
+```bash
+uv add "bilbysearch[all] @ git+https://github.com/bilbyai/bilbysearch.git"
+```
+
+## Usage
+
+```python
+import bilbysearch
+
+model = bilbysearch.load_embedder()
+
+# Embed a frame of articles. Rows with no text get a null vector rather than a
+# meaningless one, and three provenance columns record how the vectors were made.
+articles = bilbysearch.read_parquet("daily_2026-01-31.parquet")
+embedded = bilbysearch.embed_dataframe(articles, model=model)
+bilbysearch.write_parquet(embedded, "daily_2026-01-31_embedding.parquet")
+```
+
+Queries and passages are embedded by **different functions on purpose**: this model
+expects an instruction prefix on queries and none on passages. Skipping it measurably
+degrades retrieval.
+
+```python
+query = bilbysearch.embed_query("Chinese monetary policy easing", model=model)
+passages = bilbysearch.embed_texts(["The PBoC cut the reserve requirement ratio."], model=model)
+similarity = query @ passages[0]  # vectors are L2-normalised, so this is cosine
+```
+
+Only `load_embedder` needs the `[embed]` extra. Every other function works on any object
+exposing sentence-transformers' `encode`, so a caller that already has a model can pass
+it straight in.
+
+## Development
+
+```bash
+uv sync
+uv run pytest
+uv run ruff check . && uv run ruff format .
+```
+
+The tests use a stub model, so the suite runs in well under a second and never downloads
+the real 2.2 GB one.
+
+## Licence
+
+MIT — see [LICENSE](LICENSE).

bilbysearch-0.1.0/bilbysearch/__init__.py

@@ -0,0 +1,68 @@
+"""
+Embedding articles with intfloat/multilingual-e5-large-instruct.
+
+The functions callers need are re-exported here, so an importing module reads
+
+    from bilbysearch import load_embedder, embed_texts, embed_query
+
+`config.py` holds the model's identity and input contract, so every caller -
+notebooks, batch backfills, retrieval pipelines - embeds the same way.
+"""
+
+from .config import (
+    COMBINED_FIELD_SEP,
+    DEFAULT_BATCH_SIZE,
+    DEFAULT_EMBEDDINGS_COL,
+    DEFAULT_NORMALIZE_EMBEDDINGS,
+    DEFAULT_TEXT_COL,
+    EMBEDDING_DIM,
+    MAX_TOKENS,
+    MODEL_NAME,
+    OUTPUT_FILENAME_SUFFIX,
+    PROVENANCE_COLUMNS,
+    QUERY_INSTRUCTION,
+)
+from .embed import (
+    embed_dataframe,
+    embed_query,
+    embed_texts,
+    load_embedder,
+    select_output_columns,
+    select_text_series,
+)
+from .storage import (
+    ensure_parent_uri,
+    is_gcs_uri,
+    list_existing_input_dates,
+    parquet_uri,
+    read_parquet,
+    uri_exists,
+    write_parquet,
+)
+
+__all__ = [
+    "COMBINED_FIELD_SEP",
+    "DEFAULT_BATCH_SIZE",
+    "DEFAULT_EMBEDDINGS_COL",
+    "DEFAULT_NORMALIZE_EMBEDDINGS",
+    "DEFAULT_TEXT_COL",
+    "EMBEDDING_DIM",
+    "MAX_TOKENS",
+    "MODEL_NAME",
+    "OUTPUT_FILENAME_SUFFIX",
+    "PROVENANCE_COLUMNS",
+    "QUERY_INSTRUCTION",
+    "embed_dataframe",
+    "embed_query",
+    "embed_texts",
+    "ensure_parent_uri",
+    "is_gcs_uri",
+    "list_existing_input_dates",
+    "load_embedder",
+    "parquet_uri",
+    "read_parquet",
+    "select_output_columns",
+    "select_text_series",
+    "uri_exists",
+    "write_parquet",
+]

bilbysearch-0.1.0/bilbysearch/config.py

@@ -0,0 +1,49 @@
+"""
+Settings for the multilingual-e5-large-instruct embedding model.
+
+These values are the single source of truth for the model's identity and its
+input contract. Every caller - `embed.py`, notebooks, batch backfills,
+retrieval pipelines - imports them from here, so no two callers can drift apart.
+"""
+
+# HuggingFace model id. Downloaded on first use and cached in ~/.cache/huggingface.
+MODEL_NAME = "intfloat/multilingual-e5-large-instruct"
+
+# Output vector width. The Qdrant collection is created with this size, so a model
+# swap that changes it requires recreating the collection before ingesting.
+EMBEDDING_DIM = 1024
+
+# The model's context window. sentence-transformers truncates silently at this many
+# tokens; we do not truncate ourselves.
+MAX_TOKENS = 512
+
+# Queries take an instruction; passages do not. Omitting this on the query side
+# degrades retrieval quality significantly, so `embed_query` applies it and
+# `embed_texts` does not.
+QUERY_INSTRUCTION = (
+    "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
+)
+
+# Inserted between fields when a text column is "+"-separated, e.g. "title+body".
+COMBINED_FIELD_SEP = "\n\n"
+
+# Columns embedded per article in production.
+DEFAULT_TEXT_COL = "title+body"
+
+# L2-normalize vectors by default, for cosine/dot-product search. A near no-op for
+# multilingual-e5-large-instruct, whose pipeline already ends in a Normalize module.
+DEFAULT_NORMALIZE_EMBEDDINGS = True
+
+# Column embed_dataframe writes the vectors to.
+DEFAULT_EMBEDDINGS_COL = "embeddings"
+
+# Columns embed_dataframe writes beside the vectors, recording how they were made.
+# Named here so the backfill can select its output schema without restating them.
+PROVENANCE_COLUMNS = ("embedding_model", "embedding_text_col", "embedding_normalized")
+
+# Appended to the stem of backfill output files, so an embedded day is
+# daily_YYYY-MM-DD_embedding.parquet next to the daily_YYYY-MM-DD.parquet it came from.
+OUTPUT_FILENAME_SUFFIX = "_embedding"
+
+# Modest default suited to a laptop. The backfill passes a much larger value.
+DEFAULT_BATCH_SIZE = 32

bilbysearch-0.1.0/bilbysearch/embed.py

@@ -0,0 +1,306 @@
+"""
+Loading the embedding model and turning text into vectors.
+
+Documents and queries are embedded by different functions on purpose:
+multilingual-e5-large-instruct expects an instruction prefix on queries and no
+prefix on passages (see `config.QUERY_INSTRUCTION`).
+
+Only `load_embedder` needs sentence-transformers installed (`pip install
+bilbysearch[embed]`). The rest of this module works on any object exposing
+SentenceTransformer's `encode`, so a caller that already has a model can pass it in.
+"""
+
+from __future__ import annotations
+
+import logging
+
+import numpy as np
+import pandas as pd
+
+try:  # ships with the optional [embed] extra
+    from sentence_transformers import SentenceTransformer
+except ImportError:  # pragma: no cover - exercised by the blocked-import test
+    SentenceTransformer = None
+
+from .config import (
+    COMBINED_FIELD_SEP,
+    DEFAULT_BATCH_SIZE,
+    DEFAULT_EMBEDDINGS_COL,
+    DEFAULT_NORMALIZE_EMBEDDINGS,
+    DEFAULT_TEXT_COL,
+    EMBEDDING_DIM,
+    MODEL_NAME,
+    PROVENANCE_COLUMNS,
+    QUERY_INSTRUCTION,
+)
+
+
+def load_embedder(model_name: str = MODEL_NAME, device: str | None = None) -> SentenceTransformer:
+    """
+    Load the sentence-transformers embedding model.
+
+    Warns if the loaded model's output width differs from the expected 1024
+    dimensions, because a mismatch is rejected downstream by the vector store
+    rather than being caught here.
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    model_name : str, optional (default=config.MODEL_NAME)
+        HuggingFace model id, downloaded on first use and cached locally.
+    device : str or None, optional (default=None)
+        'cuda', 'mps' or 'cpu'. None (or 'auto') lets sentence-transformers choose.
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    model : SentenceTransformer
+        The loaded model, ready to pass to the embedding functions below.
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    if SentenceTransformer is None:
+        raise ImportError(
+            "load_embedder needs sentence-transformers, which ships with the optional "
+            "[embed] extra: pip install bilbysearch[embed]"
+        )
+
+    resolved_device = None if device in (None, "auto") else device
+    logging.info("Loading %s on %s", model_name, resolved_device or "auto")
+    model = SentenceTransformer(model_name, device=resolved_device)
+
+    dim = model.get_embedding_dimension()
+    if dim != EMBEDDING_DIM:
+        logging.warning(
+            "%s produces %s-d vectors, but %s-d was expected; a vector store created "
+            "for the expected width will reject these.",
+            model_name,
+            dim,
+            EMBEDDING_DIM,
+        )
+    return model
+
+
+def embed_texts(
+    texts: list[str],
+    model: SentenceTransformer,
+    batch_size: int = DEFAULT_BATCH_SIZE,
+    normalize_embeddings: bool = DEFAULT_NORMALIZE_EMBEDDINGS,
+    show_progress_bar: bool = False,
+) -> np.ndarray:
+    """
+    Embed a list of documents (no instruction prefix).
+
+    Text longer than the model's context window is truncated silently by
+    sentence-transformers at `config.MAX_TOKENS` tokens.
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    texts : list of str
+        The documents to embed. May be empty.
+    model : SentenceTransformer
+        A model from `load_embedder`.
+    batch_size : int, optional (default=config.DEFAULT_BATCH_SIZE)
+        Number of texts encoded per forward pass.
+    normalize_embeddings : bool, optional (default=config.DEFAULT_NORMALIZE_EMBEDDINGS)
+        L2-normalize the output. A no-op for multilingual-e5-large-instruct,
+        whose pipeline already ends in a Normalize module.
+    show_progress_bar : bool, optional (default=False)
+        Display a progress bar while encoding.
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    embeddings : np.ndarray
+        Array of shape (len(texts), EMBEDDING_DIM).
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    if not texts:
+        return np.empty((0, EMBEDDING_DIM), dtype="float32")
+
+    return model.encode(
+        texts,
+        batch_size=batch_size,
+        show_progress_bar=show_progress_bar,
+        convert_to_numpy=True,
+        normalize_embeddings=normalize_embeddings,
+    )
+
+
+def embed_query(
+    query: str,
+    model: SentenceTransformer,
+    instruction: str = QUERY_INSTRUCTION,
+    normalize_embeddings: bool = DEFAULT_NORMALIZE_EMBEDDINGS,
+) -> np.ndarray:
+    """
+    Embed a search query, applying the model's instruction prefix.
+
+    Use this for the query side of a search and `embed_texts` for the articles.
+    Embedding a query without the instruction measurably degrades retrieval.
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    query : str
+        The search query, without any prefix.
+    model : SentenceTransformer
+        A model from `load_embedder`.
+    instruction : str, optional (default=config.QUERY_INSTRUCTION)
+        Prefix prepended to the query before encoding.
+    normalize_embeddings : bool, optional (default=config.DEFAULT_NORMALIZE_EMBEDDINGS)
+        L2-normalize the output (a no-op for this model).
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    embedding : np.ndarray
+        A single vector of shape (EMBEDDING_DIM,).
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    embeddings = embed_texts(
+        [instruction + query],
+        model,
+        batch_size=1,
+        normalize_embeddings=normalize_embeddings,
+    )
+    return embeddings[0]
+
+
+def select_text_series(df: pd.DataFrame, text_col: str = DEFAULT_TEXT_COL) -> pd.Series:
+    """
+    Return the text to embed, joining columns when text_col is "+"-separated.
+
+    "title+body" concatenates those two columns with a blank line between them.
+    Missing values become empty strings rather than the literal "nan".
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    df : pd.DataFrame
+        Frame containing the named column(s).
+    text_col : str, optional (default=config.DEFAULT_TEXT_COL)
+        One column name, or several joined with "+".
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    texts : pd.Series
+        One string per row, aligned with df's index.
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    columns = text_col.split("+")
+    missing = [col for col in columns if col not in df.columns]
+    if missing:
+        raise ValueError(f"Input parquet is missing required text column(s): {missing}")
+
+    texts = df[columns[0]].fillna("").astype(str)
+    for col in columns[1:]:
+        texts = texts + COMBINED_FIELD_SEP + df[col].fillna("").astype(str)
+    return texts
+
+
+def embed_dataframe(
+    df: pd.DataFrame,
+    *,
+    model: SentenceTransformer,
+    model_name: str = MODEL_NAME,
+    text_col: str = DEFAULT_TEXT_COL,
+    embeddings_col: str = DEFAULT_EMBEDDINGS_COL,
+    batch_size: int = DEFAULT_BATCH_SIZE,
+    normalize_embeddings: bool = DEFAULT_NORMALIZE_EMBEDDINGS,
+    show_progress_bar: bool = True,
+) -> pd.DataFrame:
+    """
+    Add an embedding column to a frame of articles.
+
+    Rows whose combined text is empty are skipped and receive a null embedding
+    rather than a vector. Three provenance columns record how the vectors were
+    made, so an output file describes itself.
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    df : pd.DataFrame
+        Articles to embed. Not modified; a copy is returned.
+    model : SentenceTransformer
+        A model from `load_embedder`.
+    model_name : str, optional (default=config.MODEL_NAME)
+        Recorded in the embedding_model column.
+    text_col : str, optional (default=config.DEFAULT_TEXT_COL)
+        Column(s) to embed, "+"-joined. See `select_text_series`.
+    embeddings_col : str, optional (default=config.DEFAULT_EMBEDDINGS_COL)
+        Name of the column the vectors are written to.
+    batch_size : int, optional (default=config.DEFAULT_BATCH_SIZE)
+        Number of texts encoded per forward pass.
+    normalize_embeddings : bool, optional (default=config.DEFAULT_NORMALIZE_EMBEDDINGS)
+        L2-normalize the output (a no-op for this model).
+    show_progress_bar : bool, optional (default=True)
+        Display a progress bar while encoding.
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    out : pd.DataFrame
+        Copy of df with the embedding column plus embedding_model,
+        embedding_text_col and embedding_normalized.
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    out = df.copy()
+    selected = select_text_series(out, text_col)
+    valid_mask = selected.str.strip().ne("")
+    texts = selected[valid_mask].tolist()
+
+    logging.info(
+        "Embedding %s of %s rows from %r (%s skipped: empty text)",
+        len(texts),
+        len(out),
+        text_col,
+        len(out) - len(texts),
+    )
+
+    embeddings: list[list[float] | None] = [None] * len(out)
+    if texts:
+        encoded = embed_texts(
+            texts,
+            model,
+            batch_size=batch_size,
+            normalize_embeddings=normalize_embeddings,
+            show_progress_bar=show_progress_bar,
+        )
+        valid_positions = [idx for idx, is_valid in enumerate(valid_mask.tolist()) if is_valid]
+        for positional_index, vector in zip(valid_positions, encoded, strict=True):
+            embeddings[positional_index] = vector.astype("float32").tolist()
+
+    model_column, text_column, normalized_column = PROVENANCE_COLUMNS
+    out[embeddings_col] = embeddings
+    out[model_column] = model_name
+    out[text_column] = text_col
+    out[normalized_column] = normalize_embeddings
+    return out
+
+
+def select_output_columns(
+    df: pd.DataFrame,
+    *,
+    id_col: str,
+    embeddings_col: str = DEFAULT_EMBEDDINGS_COL,
+) -> pd.DataFrame:
+    """
+    Reduce an embedded frame to the columns the backfill stores.
+
+    The stored schema is the article id plus the four columns embed_dataframe adds:
+    the vectors and the three provenance columns. Everything else in the daily file
+    stays in the daily file, which remains the source of truth for article content.
+    ---------------------------------------------------------------------------
+    ARGUMENTS:
+    df : pd.DataFrame
+        A frame returned by `embed_dataframe`.
+    id_col : str
+        Column identifying the article, e.g. 'uuid'. Joins the vectors back to the
+        daily file they came from.
+    embeddings_col : str, optional (default=config.DEFAULT_EMBEDDINGS_COL)
+        Column holding the vectors.
+    ---------------------------------------------------------------------------
+    OUTPUT:
+    out : pd.DataFrame
+        Copy of df with only those five columns, in that order.
+    ---------------------------------------------------------------------------
+    AUTHORS: Samson
+    ---------------------------------------------------------------------------
+    """
+    columns = [id_col, embeddings_col, *PROVENANCE_COLUMNS]
+    missing = [col for col in columns if col not in df.columns]
+    if missing:
+        raise ValueError(f"Embedded frame is missing expected column(s): {missing}")
+    return df[columns].copy()