sentence-embedder 1.0.0__tar.gz

sentence_embedder-1.0.0/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: sentence-embedder
+Version: 1.0.0
+Summary: Lightweight Python library for sentence/semantic embeddings — SentenceTransformers & OpenAI in one unified API.
+Author-email: Your Name <you@example.com>
+License: MIT
+Project-URL: Homepage, https://github.com/your-org/sentence-embedder
+Project-URL: Repository, https://github.com/your-org/sentence-embedder
+Project-URL: Bug Tracker, https://github.com/your-org/sentence-embedder/issues
+Project-URL: Changelog, https://github.com/your-org/sentence-embedder/blob/main/CHANGELOG.md
+Keywords: embeddings,nlp,sentence-transformers,semantic-search,openai,machine-learning
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+Provides-Extra: st
+Requires-Dist: sentence-transformers>=2.7; extra == "st"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: all
+Requires-Dist: sentence-transformers>=2.7; extra == "all"
+Requires-Dist: openai>=1.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+
+# sentence-embedder
+
+A lightweight **internal Python library** for generating sentence/semantic embeddings with a clean, unified API.
+
+---
+
+## Features
+
+| Feature | Details |
+|---|---|
+| **Backends** | SentenceTransformers (local, offline) · OpenAI API |
+| **Single & batch** | `embed()` and `embed_batch()` |
+| **Similarity** | Cosine similarity between two sentences |
+| **Semantic search** | `most_similar()` over an in-memory corpus |
+| **Disk cache** | `EmbeddingCache` — skip re-embedding seen texts |
+| **Typed** | Full type hints, compatible with `mypy` |
+
+---
+
+## Installation
+
+```bash
+# Clone / copy this package into your project, then:
+
+# SentenceTransformers backend (local, recommended)
+pip install -e ".[st]"
+
+# OpenAI backend
+pip install -e ".[openai]"
+
+# Both
+pip install -e ".[all]"
+
+# With dev tools (pytest, ruff, mypy)
+pip install -e ".[all,dev]"
+```
+
+---
+
+## Quick Start
+
+### 1 — Basic embedding
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder()                 # defaults: SentenceTransformers, all-MiniLM-L6-v2
+
+vec = embedder.embed("The cat sat on the mat.")
+print(vec.shape)   # (384,)
+```
+
+### 2 — Batch embedding
+
+```python
+texts = ["Hello world", "Machine learning is fun", "I love Python"]
+vecs = embedder.embed_batch(texts)
+print(vecs.shape)  # (3, 384)
+```
+
+### 3 — Cosine similarity
+
+```python
+score = embedder.similarity("fast car", "quick automobile")
+print(score)  # ~0.85
+```
+
+### 4 — Semantic search
+
+```python
+corpus = [
+    "The stock market crashed today.",
+    "Scientists discover a new planet.",
+    "Football team wins the championship.",
+    "A new AI model beats human performance.",
+]
+
+results = embedder.most_similar("breakthrough in artificial intelligence", corpus, top_k=2)
+for sentence, score in results:
+    print(f"{score:.3f}  {sentence}")
+```
+
+### 5 — Disk cache (avoid re-embedding)
+
+```python
+from sentence_embedder import SentenceEmbedder, EmbeddingCache
+
+base = SentenceEmbedder()
+embedder = EmbeddingCache(base, cache_dir=".cache/embeddings")
+
+vec = embedder.embed("Hello world")   # computed → stored on disk
+vec = embedder.embed("Hello world")   # loaded from cache instantly
+```
+
+---
+
+## OpenAI Backend
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder(
+    backend="openai",
+    model_name="text-embedding-3-small",
+    openai_api_key="sk-...",   # or set OPENAI_API_KEY env var
+)
+
+vec = embedder.embed("Hello from OpenAI!")
+```
+
+---
+
+## API Reference
+
+### `SentenceEmbedder`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Embed one sentence → 1-D vector |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Embed many → 2-D array `(N, dim)` |
+| `similarity` | `(a: str, b: str) → float` | Cosine similarity in `[-1, 1]` |
+| `most_similar` | `(query, corpus, top_k=5) → List[tuple]` | Ranked `(sentence, score)` pairs |
+
+### `EmbeddingCache`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Cache-aware single embed |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Cache-aware batch embed |
+| `clear` | `() → None` | Wipe the cache database |
+
+---
+
+## Running Tests
+
+```bash
+pytest
+# With coverage:
+pytest --cov=sentence_embedder --cov-report=term-missing
+```
+
+---
+
+## Package Structure
+
+```
+sentence_embedder/
+├── sentence_embedder/
+│   ├── __init__.py       # Public exports
+│   ├── embedder.py       # SentenceEmbedder (core)
+│   └── cache.py          # EmbeddingCache (disk cache)
+├── tests/
+│   └── test_embedder.py  # Unit tests (no model/network needed)
+├── pyproject.toml        # Build config & dependencies
+└── README.md
+```
+
+---
+
+## Choosing a Model
+
+| Model | Dim | Speed | Quality | Use case |
+|---|---|---|---|---|
+| `all-MiniLM-L6-v2` | 384 | ⚡⚡⚡ | ★★★ | Default, general purpose |
+| `all-mpnet-base-v2` | 768 | ⚡⚡ | ★★★★ | Higher quality |
+| `multi-qa-MiniLM-L6-cos-v1` | 384 | ⚡⚡⚡ | ★★★★ | Q&A / search |
+| `text-embedding-3-small` *(OpenAI)* | 1536 | API | ★★★★★ | Best quality, needs key |

sentence_embedder-1.0.0/README.md

@@ -0,0 +1,167 @@
+# sentence-embedder
+
+A lightweight **internal Python library** for generating sentence/semantic embeddings with a clean, unified API.
+
+---
+
+## Features
+
+| Feature | Details |
+|---|---|
+| **Backends** | SentenceTransformers (local, offline) · OpenAI API |
+| **Single & batch** | `embed()` and `embed_batch()` |
+| **Similarity** | Cosine similarity between two sentences |
+| **Semantic search** | `most_similar()` over an in-memory corpus |
+| **Disk cache** | `EmbeddingCache` — skip re-embedding seen texts |
+| **Typed** | Full type hints, compatible with `mypy` |
+
+---
+
+## Installation
+
+```bash
+# Clone / copy this package into your project, then:
+
+# SentenceTransformers backend (local, recommended)
+pip install -e ".[st]"
+
+# OpenAI backend
+pip install -e ".[openai]"
+
+# Both
+pip install -e ".[all]"
+
+# With dev tools (pytest, ruff, mypy)
+pip install -e ".[all,dev]"
+```
+
+---
+
+## Quick Start
+
+### 1 — Basic embedding
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder()                 # defaults: SentenceTransformers, all-MiniLM-L6-v2
+
+vec = embedder.embed("The cat sat on the mat.")
+print(vec.shape)   # (384,)
+```
+
+### 2 — Batch embedding
+
+```python
+texts = ["Hello world", "Machine learning is fun", "I love Python"]
+vecs = embedder.embed_batch(texts)
+print(vecs.shape)  # (3, 384)
+```
+
+### 3 — Cosine similarity
+
+```python
+score = embedder.similarity("fast car", "quick automobile")
+print(score)  # ~0.85
+```
+
+### 4 — Semantic search
+
+```python
+corpus = [
+    "The stock market crashed today.",
+    "Scientists discover a new planet.",
+    "Football team wins the championship.",
+    "A new AI model beats human performance.",
+]
+
+results = embedder.most_similar("breakthrough in artificial intelligence", corpus, top_k=2)
+for sentence, score in results:
+    print(f"{score:.3f}  {sentence}")
+```
+
+### 5 — Disk cache (avoid re-embedding)
+
+```python
+from sentence_embedder import SentenceEmbedder, EmbeddingCache
+
+base = SentenceEmbedder()
+embedder = EmbeddingCache(base, cache_dir=".cache/embeddings")
+
+vec = embedder.embed("Hello world")   # computed → stored on disk
+vec = embedder.embed("Hello world")   # loaded from cache instantly
+```
+
+---
+
+## OpenAI Backend
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder(
+    backend="openai",
+    model_name="text-embedding-3-small",
+    openai_api_key="sk-...",   # or set OPENAI_API_KEY env var
+)
+
+vec = embedder.embed("Hello from OpenAI!")
+```
+
+---
+
+## API Reference
+
+### `SentenceEmbedder`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Embed one sentence → 1-D vector |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Embed many → 2-D array `(N, dim)` |
+| `similarity` | `(a: str, b: str) → float` | Cosine similarity in `[-1, 1]` |
+| `most_similar` | `(query, corpus, top_k=5) → List[tuple]` | Ranked `(sentence, score)` pairs |
+
+### `EmbeddingCache`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Cache-aware single embed |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Cache-aware batch embed |
+| `clear` | `() → None` | Wipe the cache database |
+
+---
+
+## Running Tests
+
+```bash
+pytest
+# With coverage:
+pytest --cov=sentence_embedder --cov-report=term-missing
+```
+
+---
+
+## Package Structure
+
+```
+sentence_embedder/
+├── sentence_embedder/
+│   ├── __init__.py       # Public exports
+│   ├── embedder.py       # SentenceEmbedder (core)
+│   └── cache.py          # EmbeddingCache (disk cache)
+├── tests/
+│   └── test_embedder.py  # Unit tests (no model/network needed)
+├── pyproject.toml        # Build config & dependencies
+└── README.md
+```
+
+---
+
+## Choosing a Model
+
+| Model | Dim | Speed | Quality | Use case |
+|---|---|---|---|---|
+| `all-MiniLM-L6-v2` | 384 | ⚡⚡⚡ | ★★★ | Default, general purpose |
+| `all-mpnet-base-v2` | 768 | ⚡⚡ | ★★★★ | Higher quality |
+| `multi-qa-MiniLM-L6-cos-v1` | 384 | ⚡⚡⚡ | ★★★★ | Q&A / search |
+| `text-embedding-3-small` *(OpenAI)* | 1536 | API | ★★★★★ | Best quality, needs key |

sentence_embedder-1.0.0/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sentence-embedder"
+version = "1.0.0"
+description = "Lightweight Python library for sentence/semantic embeddings — SentenceTransformers & OpenAI in one unified API."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "Your Name", email = "you@example.com" }
+]
+keywords = [
+    "embeddings", "nlp", "sentence-transformers",
+    "semantic-search", "openai", "machine-learning"
+]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+
+dependencies = [
+    "numpy>=1.24",
+]
+
+[project.urls]
+Homepage = "https://github.com/your-org/sentence-embedder"
+Repository = "https://github.com/your-org/sentence-embedder"
+"Bug Tracker" = "https://github.com/your-org/sentence-embedder/issues"
+Changelog = "https://github.com/your-org/sentence-embedder/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+st     = ["sentence-transformers>=2.7"]
+openai = ["openai>=1.0"]
+all    = ["sentence-transformers>=2.7", "openai>=1.0"]
+dev    = ["pytest>=7", "pytest-cov", "ruff", "mypy", "build", "twine"]
+
+[tool.setuptools.packages.find]
+where   = ["."]
+include = ["sentence_embedder*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts   = "-v --tb=short"
+
+[tool.ruff]
+line-length    = 88
+target-version = "py310"
+
+[tool.mypy]
+python_version      = "3.10"
+strict              = false
+ignore_missing_imports = true

sentence_embedder-1.0.0/sentence_embedder/__init__.py

@@ -0,0 +1,23 @@
+"""
+sentence_embedder
+=================
+A lightweight internal library for generating sentence/semantic embeddings.
+
+Quick start
+-----------
+>>> from sentence_embedder import SentenceEmbedder
+>>> embedder = SentenceEmbedder()                        # uses all-MiniLM-L6-v2
+>>> vec = embedder.embed("The cat sat on the mat.")
+>>> vec.shape
+(384,)
+
+>>> results = embedder.most_similar("fast car", ["racing vehicle", "slow snail", "quick automobile"])
+>>> results[0]
+('quick automobile', 0.87...)
+"""
+
+from sentence_embedder.embedder import SentenceEmbedder
+from sentence_embedder.cache import EmbeddingCache
+
+__all__ = ["SentenceEmbedder", "EmbeddingCache"]
+__version__ = "1.0.0"

sentence_embedder-1.0.0/sentence_embedder/cache.py

@@ -0,0 +1,127 @@
+"""
+cache.py
+--------
+Optional disk-based embedding cache to avoid re-computing embeddings
+for texts that have already been processed.
+
+Uses a simple ``shelve`` database keyed by ``(model_name, text)``.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import logging
+import shelve
+from pathlib import Path
+from typing import List, Optional
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+class EmbeddingCache:
+    """
+    Transparent disk cache for embedding vectors.
+
+    Wrap any :class:`~sentence_embedder.SentenceEmbedder` with this class
+    to skip re-embedding texts that were already processed.
+
+    Args:
+        embedder: A :class:`~sentence_embedder.SentenceEmbedder` instance.
+        cache_dir (str | Path): Directory where the cache database is stored.
+            Created automatically if it does not exist.
+
+    Example:
+        >>> from sentence_embedder import SentenceEmbedder
+        >>> from sentence_embedder.cache import EmbeddingCache
+        >>> base = SentenceEmbedder()
+        >>> cached = EmbeddingCache(base, cache_dir=".cache/embeddings")
+        >>> vec = cached.embed("Hello world")  # computed and stored
+        >>> vec = cached.embed("Hello world")  # retrieved from cache
+    """
+
+    def __init__(self, embedder, cache_dir: str | Path = ".cache/embeddings") -> None:
+        self._embedder = embedder
+        cache_path = Path(cache_dir)
+        cache_path.mkdir(parents=True, exist_ok=True)
+        self._db_path = str(cache_path / "embeddings")
+        logger.info("EmbeddingCache initialised at '%s'.", self._db_path)
+
+    # ------------------------------------------------------------------
+    # Public API (mirrors SentenceEmbedder)
+    # ------------------------------------------------------------------
+
+    def embed(self, text: str) -> np.ndarray:
+        """
+        Embed a single text, using the cache when available.
+
+        Args:
+            text (str): Input text.
+
+        Returns:
+            np.ndarray: Embedding vector.
+        """
+        key = self._make_key(text)
+        with shelve.open(self._db_path) as db:
+            if key in db:
+                logger.debug("Cache hit for text hash '%s'.", key)
+                return db[key]
+
+        vec = self._embedder.embed(text)
+        with shelve.open(self._db_path) as db:
+            db[key] = vec
+        return vec
+
+    def embed_batch(self, texts: List[str]) -> np.ndarray:
+        """
+        Embed a list of texts, only calling the model for cache misses.
+
+        Args:
+            texts (List[str]): Input texts.
+
+        Returns:
+            np.ndarray: 2-D array of shape ``(len(texts), dim)``.
+        """
+        keys = [self._make_key(t) for t in texts]
+        results: List[Optional[np.ndarray]] = [None] * len(texts)
+        missing_indices: List[int] = []
+
+        with shelve.open(self._db_path) as db:
+            for i, key in enumerate(keys):
+                if key in db:
+                    results[i] = db[key]
+                else:
+                    missing_indices.append(i)
+
+        if missing_indices:
+            missing_texts = [texts[i] for i in missing_indices]
+            new_vecs = self._embedder.embed_batch(missing_texts)
+            with shelve.open(self._db_path) as db:
+                for idx, vec in zip(missing_indices, new_vecs):
+                    db[keys[idx]] = vec
+                    results[idx] = vec
+
+        logger.debug(
+            "embed_batch: %d cached, %d computed.",
+            len(texts) - len(missing_indices),
+            len(missing_indices),
+        )
+        return np.stack(results)
+
+    def clear(self) -> None:
+        """Delete all cached embeddings."""
+        import os
+        for suffix in ["", ".db", ".dir", ".bak", ".dat"]:
+            path = Path(self._db_path + suffix)
+            if path.exists():
+                os.remove(path)
+        logger.info("Embedding cache cleared.")
+
+    # ------------------------------------------------------------------
+    # Helpers
+    # ------------------------------------------------------------------
+
+    def _make_key(self, text: str) -> str:
+        payload = f"{self._embedder.model_name}::{text}"
+        return hashlib.sha256(payload.encode()).hexdigest()

sentence_embedder-1.0.0/sentence_embedder/embedder.py

@@ -0,0 +1,229 @@
+"""
+embedder.py
+-----------
+Core module for generating sentence/semantic embeddings.
+Supports multiple backends: SentenceTransformers and OpenAI.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import List
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+class SentenceEmbedder:
+    """
+    A unified interface for generating sentence/semantic embeddings.
+
+    Supports the following backends:
+        - ``"sentencetransformers"`` — Local HuggingFace models (default, offline-friendly)
+        - ``"openai"`` — OpenAI text-embedding-* API models
+
+    Args:
+        backend (str): Which embedding backend to use. One of
+            ``"sentencetransformers"`` or ``"openai"``. Defaults to
+            ``"sentencetransformers"``.
+        model_name (str): Model identifier.
+            - For SentenceTransformers: any model on HuggingFace Hub,
+              e.g. ``"all-MiniLM-L6-v2"`` (default).
+            - For OpenAI: e.g. ``"text-embedding-3-small"``.
+        openai_api_key (str | None): API key for OpenAI backend. If ``None``,
+            falls back to the ``OPENAI_API_KEY`` environment variable.
+        device (str): Torch device for SentenceTransformers, e.g. ``"cpu"``
+            or ``"cuda"``. Ignored for OpenAI backend.
+
+    Example:
+        >>> embedder = SentenceEmbedder()
+        >>> vec = embedder.embed("Hello world")
+        >>> vec.shape
+        (384,)
+    """
+
+    SUPPORTED_BACKENDS = {"sentencetransformers", "openai"}
+
+    def __init__(
+        self,
+        backend: str = "sentencetransformers",
+        model_name: str = "all-MiniLM-L6-v2",
+        openai_api_key: str | None = None,
+        device: str = "cpu",
+    ) -> None:
+        backend = backend.lower()
+        if backend not in self.SUPPORTED_BACKENDS:
+            raise ValueError(
+                f"Unsupported backend '{backend}'. "
+                f"Choose from: {self.SUPPORTED_BACKENDS}"
+            )
+
+        self.backend = backend
+        self.model_name = model_name
+        self.device = device
+        self._model = None
+        self._openai_client = None
+
+        if backend == "openai":
+            self._init_openai(openai_api_key)
+        else:
+            self._init_sentence_transformers()
+
+    # ------------------------------------------------------------------
+    # Initialisation helpers
+    # ------------------------------------------------------------------
+
+    def _init_sentence_transformers(self) -> None:
+        try:
+            from sentence_transformers import SentenceTransformer
+        except ImportError as exc:
+            raise ImportError(
+                "sentence-transformers is not installed. "
+                "Run: pip install sentence-transformers"
+            ) from exc
+
+        logger.info("Loading SentenceTransformer model '%s'…", self.model_name)
+        self._model = SentenceTransformer(self.model_name, device=self.device)
+        logger.info("Model loaded.")
+
+    def _init_openai(self, api_key: str | None) -> None:
+        try:
+            from openai import OpenAI
+        except ImportError as exc:
+            raise ImportError(
+                "openai is not installed. Run: pip install openai"
+            ) from exc
+
+        import os
+
+        key = api_key or os.getenv("OPENAI_API_KEY")
+        if not key:
+            raise EnvironmentError(
+                "OpenAI API key not provided. Pass openai_api_key= or set "
+                "the OPENAI_API_KEY environment variable."
+            )
+        self._openai_client = OpenAI(api_key=key)
+        logger.info("OpenAI client initialised with model '%s'.", self.model_name)
+
+    # ------------------------------------------------------------------
+    # Public API
+    # ------------------------------------------------------------------
+
+    def embed(self, text: str) -> np.ndarray:
+        """
+        Embed a single sentence/document.
+
+        Args:
+            text (str): Input text to embed.
+
+        Returns:
+            np.ndarray: 1-D float32 embedding vector.
+
+        Raises:
+            ValueError: If ``text`` is empty.
+        """
+        if not text or not text.strip():
+            raise ValueError("Input text must not be empty.")
+        return self.embed_batch([text])[0]
+
+    def embed_batch(self, texts: List[str]) -> np.ndarray:
+        """
+        Embed a list of sentences/documents.
+
+        Args:
+            texts (List[str]): List of input strings. Empty strings are
+                rejected.
+
+        Returns:
+            np.ndarray: 2-D float32 array of shape ``(len(texts), dim)``.
+
+        Raises:
+            ValueError: If ``texts`` is empty or contains blank strings.
+        """
+        if not texts:
+            raise ValueError("texts list must not be empty.")
+        if any(not t or not t.strip() for t in texts):
+            raise ValueError("texts list must not contain empty strings.")
+
+        if self.backend == "sentencetransformers":
+            return self._embed_st(texts)
+        return self._embed_openai(texts)
+
+    def similarity(self, a: str, b: str) -> float:
+        """
+        Compute cosine similarity between two sentences.
+
+        Args:
+            a (str): First sentence.
+            b (str): Second sentence.
+
+        Returns:
+            float: Cosine similarity in [-1, 1].
+        """
+        vecs = self.embed_batch([a, b])
+        return float(_cosine_similarity(vecs[0], vecs[1]))
+
+    def most_similar(
+        self,
+        query: str,
+        corpus: List[str],
+        top_k: int = 5,
+    ) -> List[tuple[str, float]]:
+        """
+        Return the ``top_k`` most semantically similar sentences from a corpus.
+
+        Args:
+            query (str): The query sentence.
+            corpus (List[str]): Pool of candidate sentences.
+            top_k (int): Number of results to return.
+
+        Returns:
+            List[tuple[str, float]]: Ranked list of ``(sentence, score)``
+            pairs, highest similarity first.
+        """
+        if not corpus:
+            raise ValueError("corpus must not be empty.")
+        top_k = min(top_k, len(corpus))
+
+        query_vec = self.embed(query)
+        corpus_vecs = self.embed_batch(corpus)
+
+        scores = [
+            float(_cosine_similarity(query_vec, cv)) for cv in corpus_vecs
+        ]
+        ranked = sorted(zip(corpus, scores), key=lambda x: x[1], reverse=True)
+        return ranked[:top_k]
+
+    # ------------------------------------------------------------------
+    # Backend implementations
+    # ------------------------------------------------------------------
+
+    def _embed_st(self, texts: List[str]) -> np.ndarray:
+        vecs = self._model.encode(
+            texts,
+            convert_to_numpy=True,
+            show_progress_bar=len(texts) > 50,
+        )
+        return vecs.astype(np.float32)
+
+    def _embed_openai(self, texts: List[str]) -> np.ndarray:
+        response = self._openai_client.embeddings.create(
+            input=texts,
+            model=self.model_name,
+        )
+        vecs = [item.embedding for item in response.data]
+        return np.array(vecs, dtype=np.float32)
+
+
+# ------------------------------------------------------------------
+# Utilities
+# ------------------------------------------------------------------
+
+def _cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
+    """Compute cosine similarity between two vectors."""
+    norm_a = np.linalg.norm(a)
+    norm_b = np.linalg.norm(b)
+    if norm_a == 0 or norm_b == 0:
+        return 0.0
+    return float(np.dot(a, b) / (norm_a * norm_b))

sentence_embedder-1.0.0/sentence_embedder.egg-info/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: sentence-embedder
+Version: 1.0.0
+Summary: Lightweight Python library for sentence/semantic embeddings — SentenceTransformers & OpenAI in one unified API.
+Author-email: Your Name <you@example.com>
+License: MIT
+Project-URL: Homepage, https://github.com/your-org/sentence-embedder
+Project-URL: Repository, https://github.com/your-org/sentence-embedder
+Project-URL: Bug Tracker, https://github.com/your-org/sentence-embedder/issues
+Project-URL: Changelog, https://github.com/your-org/sentence-embedder/blob/main/CHANGELOG.md
+Keywords: embeddings,nlp,sentence-transformers,semantic-search,openai,machine-learning
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+Provides-Extra: st
+Requires-Dist: sentence-transformers>=2.7; extra == "st"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: all
+Requires-Dist: sentence-transformers>=2.7; extra == "all"
+Requires-Dist: openai>=1.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+
+# sentence-embedder
+
+A lightweight **internal Python library** for generating sentence/semantic embeddings with a clean, unified API.
+
+---
+
+## Features
+
+| Feature | Details |
+|---|---|
+| **Backends** | SentenceTransformers (local, offline) · OpenAI API |
+| **Single & batch** | `embed()` and `embed_batch()` |
+| **Similarity** | Cosine similarity between two sentences |
+| **Semantic search** | `most_similar()` over an in-memory corpus |
+| **Disk cache** | `EmbeddingCache` — skip re-embedding seen texts |
+| **Typed** | Full type hints, compatible with `mypy` |
+
+---
+
+## Installation
+
+```bash
+# Clone / copy this package into your project, then:
+
+# SentenceTransformers backend (local, recommended)
+pip install -e ".[st]"
+
+# OpenAI backend
+pip install -e ".[openai]"
+
+# Both
+pip install -e ".[all]"
+
+# With dev tools (pytest, ruff, mypy)
+pip install -e ".[all,dev]"
+```
+
+---
+
+## Quick Start
+
+### 1 — Basic embedding
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder()                 # defaults: SentenceTransformers, all-MiniLM-L6-v2
+
+vec = embedder.embed("The cat sat on the mat.")
+print(vec.shape)   # (384,)
+```
+
+### 2 — Batch embedding
+
+```python
+texts = ["Hello world", "Machine learning is fun", "I love Python"]
+vecs = embedder.embed_batch(texts)
+print(vecs.shape)  # (3, 384)
+```
+
+### 3 — Cosine similarity
+
+```python
+score = embedder.similarity("fast car", "quick automobile")
+print(score)  # ~0.85
+```
+
+### 4 — Semantic search
+
+```python
+corpus = [
+    "The stock market crashed today.",
+    "Scientists discover a new planet.",
+    "Football team wins the championship.",
+    "A new AI model beats human performance.",
+]
+
+results = embedder.most_similar("breakthrough in artificial intelligence", corpus, top_k=2)
+for sentence, score in results:
+    print(f"{score:.3f}  {sentence}")
+```
+
+### 5 — Disk cache (avoid re-embedding)
+
+```python
+from sentence_embedder import SentenceEmbedder, EmbeddingCache
+
+base = SentenceEmbedder()
+embedder = EmbeddingCache(base, cache_dir=".cache/embeddings")
+
+vec = embedder.embed("Hello world")   # computed → stored on disk
+vec = embedder.embed("Hello world")   # loaded from cache instantly
+```
+
+---
+
+## OpenAI Backend
+
+```python
+from sentence_embedder import SentenceEmbedder
+
+embedder = SentenceEmbedder(
+    backend="openai",
+    model_name="text-embedding-3-small",
+    openai_api_key="sk-...",   # or set OPENAI_API_KEY env var
+)
+
+vec = embedder.embed("Hello from OpenAI!")
+```
+
+---
+
+## API Reference
+
+### `SentenceEmbedder`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Embed one sentence → 1-D vector |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Embed many → 2-D array `(N, dim)` |
+| `similarity` | `(a: str, b: str) → float` | Cosine similarity in `[-1, 1]` |
+| `most_similar` | `(query, corpus, top_k=5) → List[tuple]` | Ranked `(sentence, score)` pairs |
+
+### `EmbeddingCache`
+
+| Method | Signature | Description |
+|---|---|---|
+| `embed` | `(text: str) → ndarray` | Cache-aware single embed |
+| `embed_batch` | `(texts: List[str]) → ndarray` | Cache-aware batch embed |
+| `clear` | `() → None` | Wipe the cache database |
+
+---
+
+## Running Tests
+
+```bash
+pytest
+# With coverage:
+pytest --cov=sentence_embedder --cov-report=term-missing
+```
+
+---
+
+## Package Structure
+
+```
+sentence_embedder/
+├── sentence_embedder/
+│   ├── __init__.py       # Public exports
+│   ├── embedder.py       # SentenceEmbedder (core)
+│   └── cache.py          # EmbeddingCache (disk cache)
+├── tests/
+│   └── test_embedder.py  # Unit tests (no model/network needed)
+├── pyproject.toml        # Build config & dependencies
+└── README.md
+```
+
+---
+
+## Choosing a Model
+
+| Model | Dim | Speed | Quality | Use case |
+|---|---|---|---|---|
+| `all-MiniLM-L6-v2` | 384 | ⚡⚡⚡ | ★★★ | Default, general purpose |
+| `all-mpnet-base-v2` | 768 | ⚡⚡ | ★★★★ | Higher quality |
+| `multi-qa-MiniLM-L6-cos-v1` | 384 | ⚡⚡⚡ | ★★★★ | Q&A / search |
+| `text-embedding-3-small` *(OpenAI)* | 1536 | API | ★★★★★ | Best quality, needs key |

sentence_embedder-1.0.0/sentence_embedder.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+README.md
+pyproject.toml
+sentence_embedder/__init__.py
+sentence_embedder/cache.py
+sentence_embedder/embedder.py
+sentence_embedder.egg-info/PKG-INFO
+sentence_embedder.egg-info/SOURCES.txt
+sentence_embedder.egg-info/dependency_links.txt
+sentence_embedder.egg-info/requires.txt
+sentence_embedder.egg-info/top_level.txt
+tests/test_embedder.py

sentence_embedder-1.0.0/sentence_embedder.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sentence_embedder-1.0.0/sentence_embedder.egg-info/requires.txt

@@ -0,0 +1,19 @@
+numpy>=1.24
+
+[all]
+sentence-transformers>=2.7
+openai>=1.0
+
+[dev]
+pytest>=7
+pytest-cov
+ruff
+mypy
+build
+twine
+
+[openai]
+openai>=1.0
+
+[st]
+sentence-transformers>=2.7

sentence_embedder-1.0.0/sentence_embedder.egg-info/top_level.txt

@@ -0,0 +1 @@
+sentence_embedder

sentence_embedder-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sentence_embedder-1.0.0/tests/test_embedder.py

@@ -0,0 +1,137 @@
+"""
+tests/test_embedder.py
+----------------------
+Unit tests for SentenceEmbedder and EmbeddingCache.
+Uses mocks so no model or network is required.
+"""
+
+import numpy as np
+import pytest
+from unittest.mock import MagicMock, patch
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def make_mock_st_model(dim: int = 8):
+    """Return a mock SentenceTransformer-like model."""
+    model = MagicMock()
+    model.encode = lambda texts, **kw: np.random.rand(len(texts), dim).astype(np.float32)
+    return model
+
+
+# ---------------------------------------------------------------------------
+# SentenceEmbedder — SentenceTransformers backend
+# ---------------------------------------------------------------------------
+
+class TestSentenceEmbedderST:
+    """Tests using the (mocked) SentenceTransformers backend."""
+
+    @pytest.fixture()
+    def embedder(self):
+        with patch(
+            "sentence_embedder.embedder.SentenceEmbedder._init_sentence_transformers"
+        ):
+            from sentence_embedder import SentenceEmbedder
+
+            emb = SentenceEmbedder(backend="sentencetransformers")
+            emb._model = make_mock_st_model(dim=8)
+            return emb
+
+    def test_embed_returns_1d_array(self, embedder):
+        vec = embedder.embed("Hello world")
+        assert vec.ndim == 1
+        assert vec.shape == (8,)
+
+    def test_embed_batch_returns_2d_array(self, embedder):
+        vecs = embedder.embed_batch(["Hello", "World", "Test"])
+        assert vecs.ndim == 2
+        assert vecs.shape == (3, 8)
+
+    def test_embed_empty_string_raises(self, embedder):
+        with pytest.raises(ValueError, match="empty"):
+            embedder.embed("")
+
+    def test_embed_batch_empty_list_raises(self, embedder):
+        with pytest.raises(ValueError, match="empty"):
+            embedder.embed_batch([])
+
+    def test_embed_batch_blank_string_raises(self, embedder):
+        with pytest.raises(ValueError, match="empty"):
+            embedder.embed_batch(["hello", "   "])
+
+    def test_similarity_returns_float_in_range(self, embedder):
+        score = embedder.similarity("cat", "dog")
+        assert isinstance(score, float)
+        assert -1.0 <= score <= 1.0
+
+    def test_most_similar_returns_ranked_list(self, embedder):
+        corpus = ["apple", "banana", "cherry", "date"]
+        results = embedder.most_similar("fruit", corpus, top_k=2)
+        assert len(results) == 2
+        assert all(isinstance(s, str) and isinstance(sc, float) for s, sc in results)
+
+    def test_most_similar_top_k_capped(self, embedder):
+        corpus = ["a", "b"]
+        results = embedder.most_similar("query", corpus, top_k=10)
+        assert len(results) == 2
+
+    def test_unsupported_backend_raises(self):
+        with pytest.raises(ValueError, match="Unsupported backend"):
+            from sentence_embedder import SentenceEmbedder
+            SentenceEmbedder(backend="fakegpt")
+
+
+# ---------------------------------------------------------------------------
+# EmbeddingCache
+# ---------------------------------------------------------------------------
+
+class TestEmbeddingCache:
+    """Tests for the disk cache wrapper."""
+
+    @pytest.fixture()
+    def embedder(self, tmp_path):
+        """A mock embedder that counts calls."""
+        with patch(
+            "sentence_embedder.embedder.SentenceEmbedder._init_sentence_transformers"
+        ):
+            from sentence_embedder import SentenceEmbedder, EmbeddingCache
+
+            emb = SentenceEmbedder(backend="sentencetransformers")
+            emb._model = make_mock_st_model(dim=8)
+
+            cached = EmbeddingCache(emb, cache_dir=str(tmp_path / "cache"))
+            # Track how many times the underlying model is called
+            original_embed_batch = emb.embed_batch
+            emb._call_count = 0
+
+            def counting_embed_batch(texts):
+                emb._call_count += len(texts)
+                return original_embed_batch(texts)
+
+            emb.embed_batch = counting_embed_batch
+            return emb, cached
+
+    def test_cache_miss_then_hit(self, embedder):
+        base, cached = embedder
+        _ = cached.embed("The sky is blue")
+        assert base._call_count == 1
+        _ = cached.embed("The sky is blue")
+        assert base._call_count == 1  # no second call
+
+    def test_batch_partial_cache(self, embedder):
+        base, cached = embedder
+        texts = ["sentence one", "sentence two", "sentence three"]
+        _ = cached.embed_batch(texts[:2])  # cache first two
+        base._call_count = 0              # reset counter
+        _ = cached.embed_batch(texts)     # only "sentence three" should be new
+        assert base._call_count == 1
+
+    def test_clear_cache(self, embedder, tmp_path):
+        base, cached = embedder
+        _ = cached.embed("some text")
+        cached.clear()
+        base._call_count = 0
+        _ = cached.embed("some text")    # must recompute after clear
+        assert base._call_count == 1