signalwire-sdk 3.7.2.dev133__tar.gz → 3.7.2.dev134__tar.gz

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/CHANGELOG.md

@@ -3,6 +3,16 @@
 ## [Unreleased]
 
 ### Added
+- `sw-search --query-stopwords` (and `IndexBuilder(query_stopwords=...)`)
+  records terms every document in the index is about, such as the product's
+  name. The search service and the native_vector_search skill drop them from
+  queries before embedding and keyword search, unless fewer than two content
+  words would remain, so "what is SignalWire" still searches for SignalWire.
+  In a corpus about one product, the name only favours the chunks that spell
+  it out: "SignalWire call pricing" ranked phone-number fees above call rates.
+  On SignalWire's corpus (58 labelled questions) hit@5 went from 0.88 to 0.95
+  and MRR@10 from 0.678 to 0.745; without the guard, a bare "SignalWire" fell
+  out of the top 10.
 - Chunks can carry a `rank_weight` between 0 and 1 in their metadata. Search
   multiplies it into the chunk's score, so an index can mark secondary sources,
   such as commentary indexed next to reference material, and the reference

{signalwire_sdk-3.7.2.dev133/signalwire/signalwire_sdk.egg-info → signalwire_sdk-3.7.2.dev134}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: signalwire-sdk
-Version: 3.7.2.dev133
+Version: 3.7.2.dev134
 Summary: SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation.
 Author-email: SignalWire Team <info@signalwire.com>
 License: MIT

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/docs/cli_guide.md

@@ -2325,6 +2325,7 @@ sw-search <source> [<source> ...] [options]
 - `--topic-threshold FLOAT` - Threshold for topic-based chunking (default: 0.3)
 - `--index-nlp-backend BACKEND` - NLP backend for processing: `nltk` (default) or `spacy`
 - `--embed-text {content,processed}` - What the embedding model sees: the chunk as written (`content`, default), or the keyword-search text with tags appended (`processed`), as indexes were built before this option existed
+- `--query-stopwords TERMS` - Comma-separated terms every document is about, such as your product's name. Recorded in the index; search drops them from queries, unless fewer than two content words would remain (so "what is Acme" still searches for Acme)
 - `--split-newlines N` - Split on N or more consecutive newlines, in addition to sentence boundaries
 - `--languages LANGS` - Comma-separated language codes (default: en)
 

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/docs/search_indexing.md

@@ -523,7 +523,8 @@ Common edits during review:
 
 - Add descriptive chunk IDs (replace auto-generated `chunk_0`, `chunk_1`)
 - Add relevant tags for hybrid search boosting
-- Add URLs to source documentation. In a JSON file that holds several documents, the URL (or a `document` value in the chunk's metadata) also tells search which chunks belong to the same document. Without one, every chunk in the file counts as one document and they push each other down the results.
+- Add URLs to source documentation. In a JSON file that holds several documents, the URL (or a `document` value in the chunk's metadata) also tells search which chunks belong to the same document. Without one, every chunk in the file counts as one document and they push each other down the results. Make the value identify one document: a `source` shared by the whole collection (for example `"source": "docs-site"` on every chunk) makes every file that uses it count as a single document.
+- If every document is about the same product, build with `--query-stopwords "<product name>"`. A name that every document is about carries no information in a query, and only favours the chunks that happen to spell it out. In one product corpus, "<product> call pricing" ranked the phone-number price cards (which mention porting numbers "into <product>") above the per-minute call rates.
 - Set `rank_weight` (between 0 and 1) on secondary chunks, such as commentary next to reference material, so the reference ranks first when both match
 - Add priority levels (`high`, `normal`, `low`)
 - Link related chunks via `related_chunks`

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/docs/search_overview.md

@@ -461,13 +461,14 @@ if __name__ == "__main__":
 
 When a user asks a question, the following occurs:
 
-1. The query is converted to a vector embedding.
-2. The search engine compares the query vector to all chunk vectors in the index.
-3. Chunks are scored by cosine similarity.
-4. Keyword and metadata matches boost scores (hybrid scoring).
-5. Top results (above the distance threshold) are returned.
-6. Results are formatted and provided to the LLM.
-7. The LLM generates an answer grounded in the retrieved content.
+1. If the index was built with `--query-stopwords`, those terms are removed from the query, unless fewer than two content words would remain.
+2. The query is converted to a vector embedding.
+3. The search engine compares the query vector to all chunk vectors in the index.
+4. Chunks are scored by cosine similarity.
+5. Keyword and metadata matches boost scores (hybrid scoring).
+6. Top results (above the distance threshold) are returned.
+7. Results are formatted and provided to the LLM.
+8. The LLM generates an answer grounded in the retrieved content.
 
 ### Testing Your Search
 

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "signalwire-sdk"
-version = "3.7.2.dev133"
+version = "3.7.2.dev134"
 description = "SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation."
 authors = [
     {name = "SignalWire Team", email = "info@signalwire.com"}

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/__init__.py

@@ -59,7 +59,7 @@ if TYPE_CHECKING:
     from signalwire.utils.schema_utils import SchemaValidationError
     from signalwire.web import WebService
 
-__version__ = "3.7.2.dev133"
+__version__ = "3.7.2.dev134"
 
 # §6.2-python (owner decision: NO packaging split — one package, all deps required;
 # IMPORT-TIME behavior only): the agent/web symbols are lazy-imported via module

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/agent_server.py

@@ -68,7 +68,7 @@ class AgentServer:
         self.app = FastAPI(
             title="SignalWire AI Agents",
             description="Hosted SignalWire AI Agents",
-            version="3.7.2.dev133",
+            version="3.7.2.dev134",
             redirect_slashes=False,
             docs_url=None,
             redoc_url=None,

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/cli/build_search.py

@@ -330,6 +330,17 @@ Documentation: sw-pydocs prints the SDK's installed docs, examples and API.
         ),
     )
 
+    parser.add_argument(
+        "--query-stopwords",
+        default=None,
+        help=(
+            "Comma-separated terms every document is about, such as your "
+            "product's name. Recorded in the index; search drops them from "
+            "queries unless the query would be left with fewer than two "
+            "content words."
+        ),
+    )
+
     parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
 
     parser.add_argument(
@@ -355,6 +366,13 @@ Documentation: sw-pydocs prints the SDK's installed docs, examples and API.
     return parser
 
 
+def _split_terms(value: str | None) -> list[str] | None:
+    """Split a comma-separated --query-stopwords value; None when not given."""
+    if not value:
+        return None
+    return [t.strip() for t in value.split(",") if t.strip()]
+
+
 def main() -> None:
     """Main entry point for the build-search command"""
     args = _build_parser().parse_args()
@@ -531,6 +549,7 @@ def main() -> None:
                 split_newlines=args.split_newlines,
                 index_nlp_backend=args.index_nlp_backend,
                 embed_text=args.embed_text,
+                query_stopwords=_split_terms(args.query_stopwords),
                 verbose=args.verbose,
                 semantic_threshold=args.semantic_threshold,
                 topic_threshold=args.topic_threshold,
@@ -642,6 +661,7 @@ def main() -> None:
             split_newlines=args.split_newlines,
             index_nlp_backend=args.index_nlp_backend,
             embed_text=args.embed_text,
+            query_stopwords=_split_terms(args.query_stopwords),
             verbose=args.verbose,
             semantic_threshold=args.semantic_threshold,
             topic_threshold=args.topic_threshold,

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/search/index_builder.py

@@ -57,6 +57,7 @@ class IndexBuilder:
         backend: str = "sqlite",
         connection_string: str | None = None,
         embed_text: str = "content",
+        query_stopwords: list[str] | None = None,
     ):
         """
         Initialize the index builder
@@ -96,6 +97,10 @@ class IndexBuilder:
                 'content', and 0.562 with content plus tags appended. Your
                 corpus may differ; the option exists so you can measure both.
                 Keyword search uses processed_content either way.
+            query_stopwords: Terms every document in the index is about, such
+                as the product's own name. They are recorded in the index, and
+                search drops them from queries; see
+                query_processor.drop_corpus_terms.
         """
         self.model_name = model_name
         self.chunking_strategy = chunking_strategy
@@ -111,6 +116,9 @@ class IndexBuilder:
         self.backend = backend
         self.connection_string = connection_string
         self.embed_text = embed_text
+        self.query_stopwords = [
+            t.strip() for t in (query_stopwords or []) if t and t.strip()
+        ]
         # SentenceTransformer instance, loaded lazily in _load_model(); the lib
         # ships no stubs, so this is effectively Any once populated.
         self.model: Any = None
@@ -738,6 +746,7 @@ class IndexBuilder:
                 "chunk_overlap": str(self.chunk_overlap),
                 "preprocessing_version": "1.0",
                 "embed_text": self.embed_text,
+                "query_stopwords": json.dumps(self.query_stopwords),
                 "languages": json.dumps(languages),
                 "created_at": datetime.now().isoformat(),
                 "sources": json.dumps(
@@ -962,6 +971,7 @@ class IndexBuilder:
                     "chunk_overlap": self.chunk_overlap,
                     "index_nlp_backend": self.index_nlp_backend,
                     "embed_text": self.embed_text,
+                    "query_stopwords": self.query_stopwords,
                 },
             }
 

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/search/query_processor.py

@@ -370,6 +370,97 @@ stopwords_language_map = {
     "tr": "turkish",
 }
 
+# Used when NLTK's stopword corpus is unavailable, so the guard in
+# drop_corpus_terms still recognises a question with nothing left to say.
+_FALLBACK_STOPWORDS = frozenset(
+    {
+        "a",
+        "an",
+        "and",
+        "are",
+        "as",
+        "at",
+        "be",
+        "by",
+        "can",
+        "do",
+        "does",
+        "for",
+        "from",
+        "how",
+        "i",
+        "in",
+        "is",
+        "it",
+        "its",
+        "me",
+        "my",
+        "of",
+        "on",
+        "or",
+        "the",
+        "this",
+        "that",
+        "to",
+        "vs",
+        "versus",
+        "what",
+        "when",
+        "where",
+        "which",
+        "who",
+        "why",
+        "with",
+        "you",
+        "your",
+    }
+)
+
+
+def drop_corpus_terms(
+    query: str, terms: list[str] | tuple[str, ...], language: str = "en"
+) -> str:
+    """Remove terms that every document in an index is about from a query.
+
+    In a corpus about one product, the product's name carries no information:
+    everything is about it. In the query it only rewards the few chunks that
+    happen to spell the name out. In SignalWire's own corpus, "SignalWire call
+    pricing" ranked the phone-number rate cards first, because each says
+    numbers "can be ported into SignalWire", over the per-minute rates the
+    question was about. An index declares such terms when it is built
+    (`--query-stopwords`), and search drops them here, before embedding and
+    keyword matching.
+
+    A term is kept when dropping it would leave fewer than two content words.
+    "What is SignalWire" and a bare "SignalWire" are ABOUT the name, and
+    stripping it leaves nothing to search for: measured, a bare "SignalWire"
+    fell out of the top 10 without this guard.
+
+    Matching is case-insensitive, on word boundaries, and takes a possessive
+    with it. Terms may be phrases.
+    """
+    if not isinstance(terms, (list, tuple)):
+        return query
+    clean = [t.strip() for t in terms if isinstance(t, str) and t.strip()]
+    if not query or not clean:
+        return query
+    pattern = re.compile(
+        r"(?i)\b(?:"
+        + "|".join(r"\s+".join(map(re.escape, t.split())) for t in clean)
+        + r")(?:'s)?\b"
+    )
+    if not pattern.search(query):
+        return query
+    stripped = re.sub(r"\s+", " ", pattern.sub(" ", query)).strip()
+    try:
+        stop = set(
+            nltk.corpus.stopwords.words(stopwords_language_map.get(language, "english"))
+        )
+    except Exception:  # corpus not downloaded: fall back to a small fixed list
+        stop = set(_FALLBACK_STOPWORDS)
+    content = [w for w in re.findall(r"[^\W_]+", stripped.lower()) if w not in stop]
+    return stripped if len(content) >= 2 else query
+
 
 # Function to ensure NLTK resources are downloaded
 def ensure_nltk_resources() -> None:

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/search/search_engine.py

@@ -95,6 +95,27 @@ class SearchEngine:
                 f"Invalid backend '{backend}'. Must be 'sqlite' or 'pgvector'"
             )
 
+    @property
+    def query_stopwords(self) -> list[str]:
+        """Terms the index was built to drop from queries; see drop_corpus_terms.
+
+        SQLite indexes keep them in the config table as a JSON list, pgvector
+        collections in collection_config.metadata. A missing or malformed value
+        means none, never an error: older indexes simply have no such key.
+        """
+        config: dict[str, Any] = getattr(self, "config", None) or {}
+        raw: Any = config.get("query_stopwords")
+        if raw is None and isinstance(config.get("metadata"), dict):
+            raw = config["metadata"].get("query_stopwords")
+        if isinstance(raw, str):
+            try:
+                raw = json.loads(raw)
+            except ValueError:
+                return []
+        if not isinstance(raw, list):
+            return []
+        return [t for t in raw if isinstance(t, str) and t.strip()]
+
     def _load_config(self) -> dict[str, str]:
         """Load index configuration"""
         conn = None

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/search/search_service.py

@@ -39,7 +39,7 @@ else:
     except ImportError:
         SentenceTransformer = None
 
-from .query_processor import preprocess_query, set_global_model
+from .query_processor import drop_corpus_terms, preprocess_query, set_global_model
 from .search_engine import SearchEngine
 from signalwire.core.security_config import SecurityConfig
 from signalwire.core.config_loader import ConfigLoader
@@ -564,17 +564,24 @@ class SearchService:
         # results for this query until the entry is evicted.
         degraded = False
 
+        # Terms the index declares every document is about (its product name,
+        # usually) carry no information and only reward the chunks that spell
+        # them out. See drop_corpus_terms.
+        query_text = drop_corpus_terms(
+            request.query, getattr(search_engine, "query_stopwords", None) or []
+        )
+
         # Enhance query
         try:
             enhanced = preprocess_query(
-                request.query,
+                query_text,
                 language=request.language or "auto",
                 vector=True,
                 model_name=model_name,  # Pass the correct model!
             )
         except Exception as e:
             logger.error(f"Error preprocessing query: {e}")
-            enhanced = {"enhanced_text": request.query, "vector": [], "language": "en"}
+            enhanced = {"enhanced_text": query_text, "vector": [], "language": "en"}
             degraded = True
 
         # Perform search

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134}/signalwire/signalwire/skills/native_vector_search/skill.py

@@ -691,7 +691,10 @@ class NativeVectorSearchSkill(SkillBase):
                 self.logger.debug("Remote search returned %d results", len(results))
             else:
                 # For local searches, preprocess the query locally
-                from signalwire.search.query_processor import preprocess_query
+                from signalwire.search.query_processor import (
+                    drop_corpus_terms,
+                    preprocess_query,
+                )
 
                 # Guaranteed non-None here: the guard above returns early when
                 # not use_remote and search_engine is falsy.
@@ -709,7 +712,10 @@ class NativeVectorSearchSkill(SkillBase):
                     extra_kwargs["model_name"] = model_for_query  # Use model from index
 
                 enhanced = preprocess_query(
-                    query,
+                    drop_corpus_terms(
+                        query,
+                        getattr(self.search_engine, "query_stopwords", None) or [],
+                    ),
                     language="en",
                     vector=True,
                     query_nlp_backend=self.query_nlp_backend,

{signalwire_sdk-3.7.2.dev133 → signalwire_sdk-3.7.2.dev134/signalwire/signalwire_sdk.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: signalwire-sdk
-Version: 3.7.2.dev133
+Version: 3.7.2.dev134
 Summary: SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation.
 Author-email: SignalWire Team <info@signalwire.com>
 License: MIT