signalwire-sdk 3.7.2.dev132__tar.gz → 3.7.2.dev133__tar.gz

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## [Unreleased]
+
+### Added
+- Chunks can carry a `rank_weight` between 0 and 1 in their metadata. Search
+  multiplies it into the chunk's score, so an index can mark secondary sources,
+  such as commentary indexed next to reference material, and the reference
+  ranks first when both match. It only demotes: values outside that range are
+  ignored.
+
+### Fixed
+- Search's same-file penalty now applies per document. It used to key on the
+  file a chunk was read from, so in a JSON file holding many documents every
+  chunk counted as one document and pushed the others down: in a pricing
+  collection, 50 rate cards shared one file, and the per-minute card for
+  "outbound call pricing" fell from 9th by relevance to below the top 20. A
+  chunk's document is its `document` metadata, else `url`, else `source`, else
+  its file, the same key the JSON loader already uses to tell documents apart.
+  Indexes with one document per file rank as before.
+- The penalty's diversity pass no longer raises `KeyError` on a result
+  without `filename` metadata.
+
 ## [3.7.2] - 2026-10-14
 
 ### Added

{signalwire_sdk-3.7.2.dev132/signalwire/signalwire_sdk.egg-info → signalwire_sdk-3.7.2.dev133}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: signalwire-sdk
-Version: 3.7.2.dev132
+Version: 3.7.2.dev133
 Summary: SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation.
 Author-email: SignalWire Team <info@signalwire.com>
 License: MIT

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/docs/search_indexing.md

@@ -523,7 +523,8 @@ Common edits during review:
 
 - Add descriptive chunk IDs (replace auto-generated `chunk_0`, `chunk_1`)
 - Add relevant tags for hybrid search boosting
-- Add URLs to source documentation
+- Add URLs to source documentation. In a JSON file that holds several documents, the URL (or a `document` value in the chunk's metadata) also tells search which chunks belong to the same document. Without one, every chunk in the file counts as one document and they push each other down the results.
+- Set `rank_weight` (between 0 and 1) on secondary chunks, such as commentary next to reference material, so the reference ranks first when both match
 - Add priority levels (`high`, `normal`, `low`)
 - Link related chunks via `related_chunks`
 - Remove low-value chunks (e.g., header-only content)

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/docs/search_overview.md

@@ -327,9 +327,10 @@ The algorithm proceeds in these steps:
 2. Keyword matching, weighted metadata field matching (category, product, tags, source, section, and description each carry a different weight), and filename matching run against that same pool.
 3. Each candidate is scored. If it has a vector similarity score, that score is the base, and every additional agreeing signal adds a small boost, capped at a fixed maximum. Agreement can reorder near-ties; it cannot let a weak vector match outrank a stronger one.
 4. A candidate found only through keyword or metadata matching, with no vector score, is scaled below the pool's strongest vector match. A non-semantic match can never outrank a genuine semantic one.
-5. Near-duplicate chunks are removed, keeping the highest-scoring copy of each.
-6. Results are penalized when several chunks come from the same file, so one document cannot dominate the result list. The second chunk from a file keeps 85% of its computed score, the third 70%, the fourth 50%, and the fifth or later 40%.
-7. The top N results are returned.
+5. A chunk with a `rank_weight` in its metadata has its score multiplied by it. Only values between 0 and 1 count, so a weight can demote a chunk but never promote it. Use it to mark secondary sources, such as commentary indexed next to reference material, so the reference wins when both match.
+6. Near-duplicate chunks are removed, keeping the highest-scoring copy of each.
+7. Results are penalized when several chunks come from the same document, so one document cannot dominate the result list. The second chunk of a document keeps 85% of its computed score, the third 70%, the fourth 50%, and the fifth or later 40%. A chunk's document is its `document` metadata if set, otherwise its `url`, then its `source`, then the file it was read from. Set `document` or `url` on chunks in a JSON file that holds many documents, or the whole file is treated as one.
+8. The top N results are returned.
 
 **Concrete example:**
 

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "signalwire-sdk"
-version = "3.7.2.dev132"
+version = "3.7.2.dev133"
 description = "SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation."
 authors = [
     {name = "SignalWire Team", email = "info@signalwire.com"}

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/signalwire/signalwire/__init__.py

@@ -59,7 +59,7 @@ if TYPE_CHECKING:
     from signalwire.utils.schema_utils import SchemaValidationError
     from signalwire.web import WebService
 
-__version__ = "3.7.2.dev132"
+__version__ = "3.7.2.dev133"
 
 # §6.2-python (owner decision: NO packaging split — one package, all deps required;
 # IMPORT-TIME behavior only): the agent/web symbols are lazy-imported via module

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/signalwire/signalwire/agent_server.py

@@ -68,7 +68,7 @@ class AgentServer:
         self.app = FastAPI(
             title="SignalWire AI Agents",
             description="Hosted SignalWire AI Agents",
-            version="3.7.2.dev132",
+            version="3.7.2.dev133",
             redirect_slashes=False,
             docs_url=None,
             redoc_url=None,

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/signalwire/signalwire/search/document_processor.py

@@ -95,6 +95,7 @@ else:
         magic = None
 
 from signalwire.core.logging_config import get_logger
+from signalwire.search.models import document_key
 
 logger = get_logger(__name__)
 
@@ -1796,11 +1797,7 @@ class DocumentProcessor:
                 # the heading is reset each time, and a scan of 16,894 chunks
                 # found no instance -- but that is a property of those inputs,
                 # not of this loop. Reset when the document changes.
-                doc_key = (
-                    json_metadata.get("url")
-                    or json_metadata.get("source")
-                    or json_metadata.get("filename")
-                )
+                doc_key = document_key(json_metadata)
                 if doc_key != current_doc:
                     current_doc = doc_key
                     current_heading = ""

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/signalwire/signalwire/search/models.py

@@ -8,6 +8,8 @@ See LICENSE file in the project root for full license information.
 """
 
 import warnings
+from collections.abc import Mapping
+from typing import Any
 
 # Embedding model configuration
 MODEL_ALIASES = {
@@ -40,3 +42,27 @@ def resolve_model_alias(model_name: str) -> str:
     if model_name == "large":
         warnings.warn(LARGE_ALIAS_DEPRECATION, DeprecationWarning, stacklevel=2)
     return MODEL_ALIASES.get(model_name, model_name)
+
+
+# Metadata keys naming the document a chunk came from, most specific first.
+# `document` lets an index state it outright. Without it, a page URL or a
+# repo:path source is the next-best identity, and the file the chunk was read
+# from is the last resort -- which is right for one-document-per-file indexes
+# and wrong for JSON collections, where one file holds many documents.
+DOCUMENT_KEYS = ("document", "url", "source", "filename")
+
+
+def document_key(metadata: Mapping[str, Any] | None) -> str | None:
+    """Identify the document a chunk belongs to, or None if nothing says.
+
+    Used wherever "the same document" matters: resetting heading context
+    between documents in a JSON file, and penalising several chunks of one
+    document in a result list. One definition, so the two cannot disagree.
+    """
+    if not metadata:
+        return None
+    for key in DOCUMENT_KEYS:
+        value = metadata.get(key)
+        if value:
+            return str(value)
+    return None

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133}/signalwire/signalwire/search/search_engine.py

@@ -13,6 +13,7 @@ import warnings
 from typing import Any, TYPE_CHECKING
 
 from signalwire.core.logging_config import get_logger
+from signalwire.search.models import document_key
 
 # NDArray is the runtime numpy array type when available, else Any; typed Any
 # because the fallback makes the two branches incompatible to mypy.
@@ -312,8 +313,9 @@ class SearchEngine:
           2. tag filter (optional)
           3. threshold filter on final score (redundant but safe)
           4. exact-match boost (when original_query provided)
+          4b. rank weight — the chunk's own `rank_weight` metadata, if any
           5. content dedup — collapse near-identical chunks, keep highest scorer
-          6. filename diversity — decay 2nd/3rd/4th+ chunks from the same file
+          6. document diversity — decay 2nd/3rd/4th+ chunks of one document
           7. match-type diversity — 40/40/20 hybrid/vector-only/keyword-only
           8. truncate to `count`
         """
@@ -377,6 +379,16 @@ class SearchEngine:
                         r["final_score"] = min(r["final_score"], vector_ceiling * 0.99)
             final_results.sort(key=lambda x: x["final_score"], reverse=True)
 
+        # 4b. rank weight -- see _rank_weight
+        weighted = False
+        for r in final_results:
+            w = self._rank_weight(r.get("metadata"))
+            if w != 1.0:
+                r["final_score"] *= w
+                weighted = True
+        if weighted:
+            final_results.sort(key=lambda x: x["final_score"], reverse=True)
+
         # 5. content dedup (sorted best-first, so the kept copy is the winner)
         final_results = self._dedupe_by_content(final_results)
 
@@ -1550,10 +1562,36 @@ class SearchEngine:
             deduped.append(result)
         return deduped
 
+    @staticmethod
+    def _rank_weight(metadata: dict[str, Any] | None) -> float:
+        """A chunk's declared rank weight: a number in (0, 1], else 1.0.
+
+        Lets an index say some chunks are secondary sources -- commentary next
+        to reference, say -- so that when both match, the reference wins,
+        without the query having to say which kind it wants. It multiplies the
+        final score, so it only ever demotes: nothing can be weighted above
+        what its relevance earned, which keeps the rule that non-semantic
+        signals may reorder results but not choose them. Values outside
+        (0, 1], booleans and non-numbers are ignored rather than trusted.
+        """
+        w = (metadata or {}).get("rank_weight")
+        if isinstance(w, bool) or not isinstance(w, (int, float)):
+            return 1.0
+        return float(w) if 0.0 < w <= 1.0 else 1.0
+
     def _apply_diversity_penalties(
         self, results: list[dict[str, Any]], target_count: int
     ) -> list[dict[str, Any]]:
-        """Apply penalties to prevent single-file dominance while maintaining quality"""
+        """Apply penalties to prevent single-document dominance while maintaining quality.
+
+        "Document" is document_key(): the chunk's `document`, `url` or `source`
+        metadata before its `filename`. Keying on the file alone treated every
+        chunk of a JSON collection as one document -- 50 separate rate cards in
+        one pricing file decayed each other out of the results, and the
+        per-minute card for "outbound call pricing" fell from 9th by relevance
+        to below the top 20. For one-document-per-file indexes the key is the
+        filename, as before.
+        """
         if not results:
             return results
 
@@ -1570,7 +1608,7 @@ class SearchEngine:
         }
 
         for result in results:
-            filename = (result.get("metadata") or {}).get("filename")
+            filename = document_key(result.get("metadata"))
             if filename is None:
                 # No filename signal - skip penalty logic for this result
                 penalized_result = result.copy()
@@ -1605,28 +1643,32 @@ class SearchEngine:
         # Ensure minimum diversity if we have enough results
         if len(penalized_results) > target_count:
             unique_files = len(
-                {r["metadata"]["filename"] for r in penalized_results[:target_count]}
+                {
+                    document_key(r.get("metadata"))
+                    for r in penalized_results[:target_count]
+                }
             )
 
             # If top results are too homogeneous (e.g., all from 1-2 files)
             if unique_files < min(3, target_count):
                 # Try to inject some diversity
                 selected = penalized_results[:target_count]
-                seen_files = {r["metadata"]["filename"] for r in selected}
+                seen_files = {document_key(r.get("metadata")) for r in selected}
 
                 # Look for high-quality results from other files
                 for result in penalized_results[target_count:]:
                     # Only inject a result from a new file that's reasonably good
                     # (within 50% of top score)
                     if (
-                        result["metadata"]["filename"] not in seen_files
+                        document_key(result.get("metadata")) not in seen_files
                         and result["final_score"] > 0.5 * selected[0]["final_score"]
                     ):
                         # Replace the lowest scoring result from an over-represented file
                         for i in range(len(selected) - 1, -1, -1):
-                            if file_counts[selected[i]["metadata"]["filename"]] > 2:
+                            key = document_key(selected[i].get("metadata"))
+                            if key is not None and file_counts.get(key, 0) > 2:
                                 selected[i] = result
-                                seen_files.add(result["metadata"]["filename"])
+                                seen_files.add(document_key(result.get("metadata")))
                                 break
 
                 penalized_results[:target_count] = selected

{signalwire_sdk-3.7.2.dev132 → signalwire_sdk-3.7.2.dev133/signalwire/signalwire_sdk.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: signalwire-sdk
-Version: 3.7.2.dev132
+Version: 3.7.2.dev133
 Summary: SignalWire SDK for AI voice agents, live call control (RELAY) and REST. Run sw-pydocs for its documentation.
 Author-email: SignalWire Team <info@signalwire.com>
 License: MIT