bgts-context-engine 0.2.2__tar.gz → 0.2.3__tar.gz

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/CHANGELOG.md

@@ -7,6 +7,56 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.2.3] - 2026-09-21
+
+### Added
+
+- `BCE_EMBEDDING_PROVIDER=openai`: an encoder that talks to any server speaking the OpenAI
+  `/v1/embeddings` protocol (vLLM, Text Embeddings Inference, Ollama, or OpenAI itself), using
+  only the standard library. This is how an open model such as `jinaai/jina-code-embeddings-1.5b`
+  runs on-prem. Query/document asymmetry is a text prefix — the jina-code family's `nl2code`
+  prompts are the default when the model name contains `jina-code`; anything else is unset
+  unless `BCE_EMBEDDING_QUERY_PREFIX` / `BCE_EMBEDDING_DOCUMENT_PREFIX` say otherwise.
+  `BCE_EMBEDDING_BASE_URL`, `BCE_EMBEDDING_API_KEY`, `BCE_EMBEDDING_EXTRA_BODY` (default
+  `{"truncate_prompt_tokens": -1}` so vLLM truncates instead of rejecting a long batch; set
+  `{}` for OpenAI's own API, which rejects unknown fields) and `BCE_EMBEDDING_TIMEOUT` (600 s)
+  configure the client. Vectors wider than `BCE_EMBEDDING_DIM` are truncated and re-normalised
+  (Matryoshka); a narrower reply is a configuration error. Transient 5xx / connection failures
+  retry three times; 4xx does not.
+- Retrieval profiles (`bce.core.orchestrator.profile`): the three engine constants that encode
+  *where in the model's ranked list the right answers sit* — semantic-anchor rank scale, the
+  share of the answer reserved for the model's ranks, and a guard on its top ranks in
+  narrowing — grouped per embedding family. `BCE_RETRIEVAL_PROFILE=auto` (the default) picks
+  `voyage` for `voyage-*` and unknown models (the historical constants, unchanged) and `jina`
+  for `jina-*`. Name one to force it; `BCE_RETRIEVAL_SEMANTIC_GUARD_RANKS` /
+  `_RESERVE_SHARE` / `_RANK_SCALE` override a single knob for fitting runs. A profile change
+  never needs a re-index. The active profile is written into every `bce bench-prs` report.
+- `bce migrate --reset-embeddings`: after the SQL migrations, `align_embedding_dim` re-types
+  `embeddings.embedding` to `BCE_EMBEDDING_DIM`. Stored vectors of another width are refused
+  unless the flag is set (they belong to another model and would be discarded); then the HNSW
+  index is rebuilt the same way migration `0005` did.
+
+### Changed
+
+- Narrowing reads the active retrieval profile. The voyage path is unchanged (no guard,
+  share 0.5). The jina profile adds a guard of 10: the model's first ten ranks fill slots
+  1–10 (only the container cap can skip one). jina-code finds more correct symbols than
+  voyage-code-4 but spreads them over ranks 1–15, and under the voyage constants the engine's
+  agreement with the model's *deeper* ranks satisfied the reserve share and pushed those mid
+  hits to positions 11–19 or out of the answer (PR replay, 30 PRs: recall@20 35.7 % → 38.2 %,
+  holdout 27.8 % → 34.9 %, retention of the model's own hits 83 % → 92 %). The list stays
+  K-monotonic.
+- `bce bench-prs` titles the markdown with the encoder that actually produced the snapshots
+  (`jina-code-embeddings-1.5b vs … + BCE`), because the JSON key `voyage` is historical, not
+  the model.
+
+### Fixed
+
+- `bce bench-prs --from-report` / `--rerun-bce` no longer treats a stored semantic pool
+  shorter than K as incomplete. A task whose neighbours are mostly tests legitimately has a
+  pool shorter than K — the live run had nothing more either — so the check is now "non-empty
+  and at least as long as the list scored from it".
+
 ## [0.2.2] - 2026-09-18
 
 ### Added
@@ -254,7 +304,8 @@ First public release.
 - Docker Compose deployment with PostgreSQL, Apache AGE and pgvector preconfigured.
 - `server.json` manifest describing the stdio server for the official MCP registry.
 
-[Unreleased]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.2.2...HEAD
+[Unreleased]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.2.3...HEAD
+[0.2.3]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.2.2...v0.2.3
 [0.2.2]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.2.1...v0.2.2
 [0.2.1]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/bgts-ai-org/bgts-context-engine/compare/v0.1.0...v0.2.0

{bgts_context_engine-0.2.2/src/bgts_context_engine.egg-info → bgts_context_engine-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bgts-context-engine
-Version: 0.2.2
+Version: 0.2.3
 Summary: Deterministic code-graph context engine for AI coding agents - PostgreSQL, Apache AGE and pgvector, over MCP and REST
 Author: Oğuz Öztürk, Enes İyidil
 Maintainer: BGTS
@@ -69,7 +69,7 @@ that actually answer it — ranked, budgeted, and reproducible.
 [![MCP](https://img.shields.io/badge/MCP-compatible-000000.svg)](docs/mcp.md)
 [![Stars](https://img.shields.io/github/stars/bgts-ai-org/bgts-context-engine?style=flat&logo=github)](https://github.com/bgts-ai-org/bgts-context-engine/stargazers)
 
-[Quick start](#quick-start) · [Use it from your agent](#use-it-from-your-agent) · [How it works](#how-it-works) · [Documentation](#documentation) · [Türkçe](README.tr.md)
+[Quick start](#quick-start) · [Use it from your agent](#use-it-from-your-agent) · [How it works](#how-it-works) · [Supported models](#supported-models) · [Documentation](#documentation) · [Türkçe](README.tr.md)
 
 </div>
 
@@ -294,7 +294,36 @@ The full formula, every weight, and the confidence thresholds are in
 - **A UI that explains itself.** `/ui` ships in the wheel and replays a real retrieval call
   stage by stage: anchors lighting up, expansion spreading, candidates scored and cut.
 - **Runs offline.** The default embedding provider is deterministic arithmetic over token
-  digests. No API key, no network, repeatable benchmarks.
+  digests. No API key, no network, repeatable benchmarks. `openai` talks to any
+  OpenAI-compatible `/v1/embeddings` server (vLLM, TEI, Ollama), so a model such as
+  [jina-code-embeddings-1.5b](https://huggingface.co/jinaai/jina-code-embeddings-1.5b)
+  can run inside the perimeter.
+
+## Supported models
+
+Embeddings only find entry points when the task text names nothing the graph already
+knows. They never rank the answer. Out of the box that seed is `hashing`: deterministic
+arithmetic, no API key, no network. For a real code model, set `BCE_EMBEDDING_PROVIDER`
+and `BCE_EMBEDDING_MODEL` to one of these:
+
+| Model | Provider | Dimension |
+| --- | --- | --- |
+| [`voyage-code-3`](https://blog.voyageai.com/2024/12/04/voyage-code-3/) | Voyage AI (`voyage`) | 1024 |
+| [`voyage-code-4`](https://blog.voyageai.com/2026/08/13/voyage-code-4/) | Voyage AI (`voyage`) | 1024 |
+| [`jina-code-embeddings-1.5b`](https://huggingface.co/jinaai/jina-code-embeddings-1.5b) | OpenAI-compatible (`openai`) | 1536 |
+
+Voyage is a hosted API — `pip install "bgts-context-engine[embed]"` and
+`BCE_VOYAGE_API_KEY`. Jina is the on-prem path: any server that speaks `/v1/embeddings`
+(vLLM, TEI, Ollama). Switching the model or the dimension is a re-index
+(`bce migrate --reset-embeddings`). The knobs are in
+[docs/deployment.md](docs/deployment.md).
+
+**Coming next** — same `openai` socket, not yet a fitted retrieval profile:
+
+- [`jina-code-embeddings-0.5b`](https://huggingface.co/jinaai/jina-code-embeddings-0.5b)
+  — the smaller sibling of 1.5b, for hosts that cannot hold 1.5B parameters.
+- [`Nomic Embed Code`](https://huggingface.co/nomic-ai/nomic-embed-code) — an open 7B
+  code retriever.
 
 ## Where it fits
 

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/README.md

@@ -16,7 +16,7 @@ that actually answer it — ranked, budgeted, and reproducible.
 [![MCP](https://img.shields.io/badge/MCP-compatible-000000.svg)](docs/mcp.md)
 [![Stars](https://img.shields.io/github/stars/bgts-ai-org/bgts-context-engine?style=flat&logo=github)](https://github.com/bgts-ai-org/bgts-context-engine/stargazers)
 
-[Quick start](#quick-start) · [Use it from your agent](#use-it-from-your-agent) · [How it works](#how-it-works) · [Documentation](#documentation) · [Türkçe](README.tr.md)
+[Quick start](#quick-start) · [Use it from your agent](#use-it-from-your-agent) · [How it works](#how-it-works) · [Supported models](#supported-models) · [Documentation](#documentation) · [Türkçe](README.tr.md)
 
 </div>
 
@@ -241,7 +241,36 @@ The full formula, every weight, and the confidence thresholds are in
 - **A UI that explains itself.** `/ui` ships in the wheel and replays a real retrieval call
   stage by stage: anchors lighting up, expansion spreading, candidates scored and cut.
 - **Runs offline.** The default embedding provider is deterministic arithmetic over token
-  digests. No API key, no network, repeatable benchmarks.
+  digests. No API key, no network, repeatable benchmarks. `openai` talks to any
+  OpenAI-compatible `/v1/embeddings` server (vLLM, TEI, Ollama), so a model such as
+  [jina-code-embeddings-1.5b](https://huggingface.co/jinaai/jina-code-embeddings-1.5b)
+  can run inside the perimeter.
+
+## Supported models
+
+Embeddings only find entry points when the task text names nothing the graph already
+knows. They never rank the answer. Out of the box that seed is `hashing`: deterministic
+arithmetic, no API key, no network. For a real code model, set `BCE_EMBEDDING_PROVIDER`
+and `BCE_EMBEDDING_MODEL` to one of these:
+
+| Model | Provider | Dimension |
+| --- | --- | --- |
+| [`voyage-code-3`](https://blog.voyageai.com/2024/12/04/voyage-code-3/) | Voyage AI (`voyage`) | 1024 |
+| [`voyage-code-4`](https://blog.voyageai.com/2026/08/13/voyage-code-4/) | Voyage AI (`voyage`) | 1024 |
+| [`jina-code-embeddings-1.5b`](https://huggingface.co/jinaai/jina-code-embeddings-1.5b) | OpenAI-compatible (`openai`) | 1536 |
+
+Voyage is a hosted API — `pip install "bgts-context-engine[embed]"` and
+`BCE_VOYAGE_API_KEY`. Jina is the on-prem path: any server that speaks `/v1/embeddings`
+(vLLM, TEI, Ollama). Switching the model or the dimension is a re-index
+(`bce migrate --reset-embeddings`). The knobs are in
+[docs/deployment.md](docs/deployment.md).
+
+**Coming next** — same `openai` socket, not yet a fitted retrieval profile:
+
+- [`jina-code-embeddings-0.5b`](https://huggingface.co/jinaai/jina-code-embeddings-0.5b)
+  — the smaller sibling of 1.5b, for hosts that cannot hold 1.5B parameters.
+- [`Nomic Embed Code`](https://huggingface.co/nomic-ai/nomic-embed-code) — an open 7B
+  code retriever.
 
 ## Where it fits
 

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/README.tr.md

@@ -14,7 +14,7 @@ veren sekiz sembolü sıralanmış, bütçelenmiş ve yeniden üretilebilir şek
 [![MCP](https://img.shields.io/badge/MCP-uyumlu-000000.svg)](docs/mcp.md)
 [![Yıldızlar](https://img.shields.io/github/stars/bgts-ai-org/bgts-context-engine?style=flat&logo=github)](https://github.com/bgts-ai-org/bgts-context-engine/stargazers)
 
-[Hızlı başlangıç](#hızlı-başlangıç) · [Ajanınızdan kullanma](#ajanınızdan-kullanma) · [Nasıl çalışır](#nasıl-çalışır) · [Dokümantasyon](#dokümantasyon) · [English](README.md)
+[Hızlı başlangıç](#hızlı-başlangıç) · [Ajanınızdan kullanma](#ajanınızdan-kullanma) · [Nasıl çalışır](#nasıl-çalışır) · [Desteklenen modeller](#desteklenen-modeller) · [Dokümantasyon](#dokümantasyon) · [English](README.md)
 
 </div>
 
@@ -245,7 +245,36 @@ Formülün tamamı, her ağırlık ve güven eşikleri
   aşama aşama oynatır: çapaların yanması, genişlemenin yayılması, adayların skorlanıp
   kesilmesi.
 - **Çevrimdışı çalışır.** Varsayılan embedding sağlayıcısı, token özetleri üzerinde
-  deterministik aritmetiktir. API anahtarı yok, ağ yok, tekrarlanabilir ölçümler.
+  deterministik aritmetiktir. API anahtarı yok, ağ yok, tekrarlanabilir ölçümler. `openai`
+  sağlayıcısı OpenAI uyumlu herhangi bir `/v1/embeddings` sunucusuna konuşur (vLLM, TEI,
+  Ollama); [jina-code-embeddings-1.5b](https://huggingface.co/jinaai/jina-code-embeddings-1.5b)
+  gibi bir model çevre içinde çalışır.
+
+## Desteklenen modeller
+
+Embedding yalnızca grafın tanımadığı bir görev metninde giriş noktası bulur. Cevabı asla
+sıralamaz. Varsayılan `hashing`'dir: deterministik aritmetik, API anahtarı yok, ağ yok.
+Gerçek bir kod modeli için `BCE_EMBEDDING_PROVIDER` ve `BCE_EMBEDDING_MODEL`'i şunlardan
+birine ayarlayın:
+
+| Model | Sağlayıcı | Boyut |
+| --- | --- | --- |
+| [`voyage-code-3`](https://blog.voyageai.com/2024/12/04/voyage-code-3/) | Voyage AI (`voyage`) | 1024 |
+| [`voyage-code-4`](https://blog.voyageai.com/2026/08/13/voyage-code-4/) | Voyage AI (`voyage`) | 1024 |
+| [`jina-code-embeddings-1.5b`](https://huggingface.co/jinaai/jina-code-embeddings-1.5b) | OpenAI uyumlu (`openai`) | 1536 |
+
+Voyage barındırılan bir API'dir — `pip install "bgts-context-engine[embed]"` ve
+`BCE_VOYAGE_API_KEY`. Jina çevre-içi yoldur: `/v1/embeddings` konuşan herhangi bir
+sunucu (vLLM, TEI, Ollama). Model veya boyutu değiştirmek yeniden indekslemedir
+(`bce migrate --reset-embeddings`). Ayarlar
+[docs/deployment.md](docs/deployment.md) içindedir.
+
+**Sırada** — aynı `openai` soketi, henüz ayrı bir getirme profili yok:
+
+- [`jina-code-embeddings-0.5b`](https://huggingface.co/jinaai/jina-code-embeddings-0.5b)
+  — 1.5b'nin küçük kardeşi; 1.5B parametreyi taşıyamayan makineler için.
+- [`Nomic Embed Code`](https://huggingface.co/nomic-ai/nomic-embed-code) — açık kaynaklı
+  7B kod getiricisi.
 
 ## Nereye oturur
 

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/__init__.py

@@ -5,4 +5,4 @@ specification for details. This package is the provider side of the deterministi
 contains no LLM and produces reproducible context packages.
 """
 
-__version__ = "0.2.2"
+__version__ = "0.2.3"

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/api/rest/ui/trace.py

@@ -21,6 +21,7 @@ from bce.core.coverage import compute_coverage
 from bce.core.orchestrator.anchors import find_anchors
 from bce.core.orchestrator.expand import expand_from_anchors, to_candidates
 from bce.core.orchestrator.orchestrator import RetrievalResult, narrow
+from bce.core.orchestrator.profile import active_profile
 from bce.core.orchestrator.text import content_tokens
 from bce.core.scoring.engine import ScoreWeights, score_candidates
 from bce.storage.graph.repository import GraphRepository
@@ -69,6 +70,7 @@ def trace_context_for_task(
 
     # Stage 1: multi-source anchors (spec section 6.2), with evidence strength per anchor.
     t = time.perf_counter()
+    profile = active_profile()
     anchors = find_anchors(
         repository,
         task_text=task_text,
@@ -78,6 +80,7 @@ def trace_context_for_task(
         component_repo_ids=None,
         repo_ids=repo_ids,
         semantic_candidates=semantic_candidates,
+        profile=profile,
     )
     stages.append(
         {
@@ -139,7 +142,7 @@ def trace_context_for_task(
 
     # Stage 4: diversity-aware narrowing to top-N (the "1000 -> 8" step).
     t = time.perf_counter()
-    narrowed = narrow(ranked, max_candidates)
+    narrowed = narrow(ranked, max_candidates, profile=profile)
     stages.append(
         {
             "stage": "narrow",

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/bench/pr_ablation.py

@@ -666,6 +666,9 @@ class AblationReport:
     cases: list[CaseOutcome]
     #: Extra K columns derived from the K-long lists (see :func:`score_runs`).
     ks: list[int] = field(default_factory=list)
+    #: The retrieval profile BCE ran with (name + the model-specific constants), so two runs of
+    #: the same snapshots can be told apart by their engine tuning.
+    profile: dict[str, Any] = field(default_factory=dict)
 
     def _rows(
         self, variant: str, *, scored_only: bool, split: str | None, level: str = "symbol"
@@ -848,6 +851,7 @@ class AblationReport:
             "k": self.k,
             "ks": list(self.ks),
             "encoder_model": self.encoder_model,
+            "profile": dict(self.profile),
             "variants": list(self.variants),
             "summary": {v: self.aggregate(v) for v in self.variants},
             "summary_by_split": {
@@ -1124,10 +1128,14 @@ def run_ablation(
                     for system in SYSTEMS
                     if replay is not None and (case.spec.number, variant, system) in replay
                 }
+                # The stored pool is complete when it is at least as long as the list that was
+                # scored from it: a task whose neighbours are mostly tests legitimately has a
+                # pool shorter than K (the live run had nothing more either).
                 baselines_ok = (
                     "voyage" in replayed
                     and "voyage_raw" in replayed
-                    and len(replayed["voyage"].semantic_pool) >= k
+                    and bool(replayed["voyage"].semantic_pool)
+                    and len(replayed["voyage"].semantic_pool) >= len(replayed["voyage"].returned)
                 )
                 if len(replayed) == len(SYSTEMS) and not rerun_bce:
                     runs = replayed
@@ -1157,8 +1165,15 @@ def run_ablation(
                 outcomes.append(outcome)
                 if progress is not None:
                     progress(outcome)
+    from bce.core.orchestrator.profile import active_profile
+
     return AblationReport(
-        k=k, encoder_model=encoder_model, variants=list(variants), cases=outcomes, ks=list(ks)
+        k=k,
+        encoder_model=encoder_model,
+        variants=list(variants),
+        cases=outcomes,
+        ks=list(ks),
+        profile=asdict(active_profile()),
     )
 
 
@@ -1355,9 +1370,15 @@ def _render_text_buckets(lines: list[str], buckets: dict[str, Any]) -> None:
 
 def render_markdown(report: AblationReport) -> str:
     lines: list[str] = []
-    lines.append("# Voyage vs Voyage + BCE - PR replay benchmark")
+    # The pure-embedding baseline is keyed ``voyage`` in the JSON for historical reasons; the
+    # actual model is whatever encoded the snapshots, so the title says that instead.
+    model_name = report.encoder_model.rpartition("-")[0] or report.encoder_model
+    lines.append(f"# {model_name} vs {model_name} + BCE - PR replay benchmark")
     lines.append("")
     lines.append(f"- Encoder: `{report.encoder_model}` (same embedding rows for both systems)")
+    if report.profile:
+        knobs = ", ".join(f"{k}={v}" for k, v in report.profile.items() if k != "name")
+        lines.append(f"- Retrieval profile: `{report.profile.get('name', '?')}` ({knobs})")
     lines.append(f"- K (results per system): {report.k}")
     scored = sorted({c.number for c in report.cases if c.scored})
     unscored = sorted({c.number for c in report.cases if not c.scored})
@@ -1370,8 +1391,9 @@ def render_markdown(report: AblationReport) -> str:
     lines.append("")
     lines.append(
         "Ground truth = symbols of the pre-PR snapshot that the PR's diff touched (tests and "
-        "files created by the PR excluded). `voyage` = nearest non-test symbol embeddings; "
-        "`voyage_raw` = unfiltered nearest neighbours; `bce` = full engine with the same "
+        f"files created by the PR excluded). `voyage` = pure `{model_name}` search, nearest "
+        f"non-test symbol embeddings (the key is historical, not the model); `voyage_raw` = "
+        "unfiltered nearest neighbours; `bce` = full engine with the same "
         "neighbours as semantic anchors. Retention = share of `voyage`'s correct hits the system "
         "still returns. Engine constants were fitted on `tune` only, so `holdout` is the honest "
         "number."

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/cli.py

@@ -69,15 +69,39 @@ def _root(
 
 
 @app.command()
-def migrate() -> None:
-    """Apply pending database migrations."""
-    from bce.storage.relational.migrator import run_migrations
+def migrate(
+    reset_embeddings: bool = typer.Option(
+        False,
+        "--reset-embeddings",
+        help=(
+            "Allow re-typing embeddings.embedding to BCE_EMBEDDING_DIM even when it holds vectors "
+            "of another width (they are discarded; reindex afterwards)"
+        ),
+    ),
+) -> None:
+    """Apply pending database migrations and fit the vector column to BCE_EMBEDDING_DIM."""
+    from bce.config import get_settings
+    from bce.storage.relational.db import connection
+    from bce.storage.relational.migrator import (
+        EmbeddingDimMismatch,
+        align_embedding_dim,
+        run_migrations,
+    )
 
-    applied = run_migrations()
+    dim = get_settings().embedding_dim
+    with connection() as conn:
+        applied = run_migrations(conn)
+        try:
+            resized = align_embedding_dim(conn, dim, reset=reset_embeddings)
+        except EmbeddingDimMismatch as exc:
+            typer.echo(f"Configuration error: {exc}", err=True)
+            raise typer.Exit(code=1) from exc
     if applied:
         typer.echo("Applied migrations: " + ", ".join(applied))
     else:
         typer.echo("No pending migrations.")
+    if resized:
+        typer.echo(f"Re-typed embeddings.embedding to vector({dim}).")
 
 
 @app.command()

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/config.py

@@ -13,6 +13,7 @@ from __future__ import annotations
 
 import os
 from functools import lru_cache
+from typing import Any
 
 from pydantic import Field
 from pydantic_settings import BaseSettings, SettingsConfigDict
@@ -61,19 +62,54 @@ class Settings(BaseSettings):
     cors_origins: tuple[str, ...] = ("http://localhost:5173", "http://127.0.0.1:5173")
 
     # --- Embedding (Phase 2; pinned for determinism, P2) ---
-    # Provider selects the encoder: "hashing" (default, dependency-free, deterministic fallback) or
-    # "voyage" (Voyage AI code embeddings). Model + dim are pinned; a change is a reindex boundary.
+    # Provider selects the encoder: "hashing" (default, dependency-free, deterministic fallback),
+    # "voyage" (Voyage AI code embeddings) or "openai" (any OpenAI-compatible ``/v1/embeddings``
+    # server: vLLM, TEI, Ollama, OpenAI itself - the way to run an open model such as
+    # jinaai/jina-code-embeddings-1.5b). Model + dim are pinned; a change is a reindex boundary.
     embedding_provider: str = Field(
         default="hashing",
-        description="Embedding backend: 'hashing' (local fallback) or 'voyage' (Voyage AI).",
+        description=(
+            "Embedding backend: 'hashing' (local fallback), 'voyage' (Voyage AI) or 'openai' "
+            "(OpenAI-compatible HTTP endpoint)."
+        ),
     )
     embedding_model: str = Field(
         default="voyage-code-3",
         description="Pinned embedding model identifier. Versioned for reproducibility (P2).",
     )
+    # Width of the pgvector column. ``bce migrate`` re-types ``embeddings.embedding`` to this width;
+    # an encoder returning wider vectors is truncated to it and re-normalised (Matryoshka models).
     embedding_dim: int = 1024
     # Voyage AI API key (used only when embedding_provider == 'voyage').
     voyage_api_key: str = ""
+    # --- provider "openai" only ---
+    # Base URL of the OpenAI-compatible server; ``/embeddings`` is appended.
+    embedding_base_url: str = "http://127.0.0.1:8001/v1"
+    # Bearer token; leave empty for a local server that does not check one.
+    embedding_api_key: str = ""
+    # Instruction prefixes prepended to queries / documents (instruction-tuned models). ``None``
+    # picks a built-in default from the model name (jina-code-* -> its nl2code prompts); set a
+    # string - even an empty one - to override.
+    embedding_query_prefix: str | None = None
+    embedding_document_prefix: str | None = None
+    # Extra JSON fields merged into every request body. The default asks vLLM to truncate inputs
+    # longer than its ``--max-model-len`` instead of rejecting the batch; other servers ignore the
+    # field, OpenAI's own API rejects unknown fields, so set it to ``{}`` there.
+    embedding_extra_body: dict[str, Any] = Field(
+        default_factory=lambda: {"truncate_prompt_tokens": -1}
+    )
+    # Seconds to wait for one embedding request (a local model on a laptop can need minutes).
+    embedding_timeout: float = 600.0
+
+    # --- Retrieval profile (engine constants that depend on the embedding model) ---
+    # "auto" (default) picks the profile fitted for ``embedding_model`` - "voyage" for voyage-*,
+    # "jina" for jina-*, voyage's constants for anything else; or name one explicitly. The three
+    # optional knobs override single values of the chosen profile (fitting runs; leave unset).
+    # See ``bce.core.orchestrator.profile``.
+    retrieval_profile: str = "auto"
+    retrieval_semantic_guard_ranks: int | None = None
+    retrieval_semantic_reserve_share: float | None = None
+    retrieval_semantic_rank_scale: float | None = None
 
     # --- Logging (JSON to stdout; optional rotating file sink) ---
     log_level: str = "INFO"

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/core/orchestrator/anchors.py

@@ -37,6 +37,7 @@ import re
 from dataclasses import dataclass, field
 
 from bce.core.orchestrator import bulk
+from bce.core.orchestrator.profile import RetrievalProfile
 from bce.core.orchestrator.text import (
     explicit_references,
     is_test_symbol,
@@ -86,6 +87,8 @@ _SATURATED_TERM_WEIGHT = 0.35
 #: ``rank / len(list)`` form made rank 29 of a 30-list as strong as rank 10 of an 11-list).
 #: Measured on the PR-replay tune split: 8 / 10 / 15 all lost semantic retention (80 %) and
 #: recall@20 against 30, which keeps every model rank in the pool; steeper decay is not paid for.
+#: This is the voyage-code-4 value; another embedding model may carry its own in its
+#: :class:`~bce.core.orchestrator.profile.RetrievalProfile` (``semantic_rank_scale``).
 SEMANTIC_RANK_SCALE = 30
 
 
@@ -144,11 +147,13 @@ def find_anchors(
     lexical_terms: list[str] | None = None,
     semantic_candidates: list[str] | None = None,
     include_tests: bool = False,
+    profile: RetrievalProfile | None = None,
 ) -> AnchorResult:
     """Combine the four anchor sources deterministically. Callers supply pre-fetched metadata.
 
     ``semantic_candidates`` must be in the model's rank order (best first); rank decides the
     semantic evidence strength. ``include_tests`` admits test symbols from lexical/semantic.
+    ``profile`` supplies the model-specific rank scale (default: the voyage constants).
     """
     result = AnchorResult()
     for rid in sorted(set(component_repo_ids or [])):
@@ -181,7 +186,10 @@ def find_anchors(
         result.add(sid, "lexical", strength)
 
     # Source 4b: semantic candidates (lowest priority; caller supplies the model's ranked list).
-    for sid, rank, strength in _semantic_anchors(repository, semantic_candidates, include_tests):
+    scale = profile.semantic_rank_scale if profile is not None else SEMANTIC_RANK_SCALE
+    for sid, rank, strength in _semantic_anchors(
+        repository, semantic_candidates, include_tests, scale=scale
+    ):
         result.add(sid, "semantic", strength)
         result.semantic_rank[sid] = rank
 
@@ -246,11 +254,13 @@ def _semantic_anchors(
     repository: GraphRepository,
     semantic_candidates: list[str] | None,
     include_tests: bool,
+    *,
+    scale: float = SEMANTIC_RANK_SCALE,
 ) -> list[tuple[str, int, float]]:
     """``(symbol_id, rank, strength)`` for the model's ranked list, strongest first.
 
     Strength decays hyperbolically with the absolute rank - ``cap / (1 + rank / scale)``
-    (:data:`SEMANTIC_RANK_SCALE`) - so the best hit is worth as much as a moderately ambiguous
+    (:data:`SEMANTIC_RANK_SCALE` unless the profile says otherwise) - so the best hit is worth as much as a moderately ambiguous
     explicit name, mid ranks still expand, and the tail of a wide pool is kept as candidates only.
     Only the rank enters the engine: a similarity number would let the model reorder candidates,
     which is the one thing the deterministic core does not delegate.
@@ -265,13 +275,13 @@ def _semantic_anchors(
             info = meta.get(sid) or {}
             if is_test_symbol(sid, info.get("name"), info.get("file_id")):
                 continue
-        out.append((sid, rank, semantic_strength(rank)))
+        out.append((sid, rank, semantic_strength(rank, scale=scale)))
     return out
 
 
-def semantic_strength(rank: int) -> float:
+def semantic_strength(rank: int, *, scale: float = SEMANTIC_RANK_SCALE) -> float:
     """Anchor strength for the model's 0-based ``rank`` (see :data:`SEMANTIC_RANK_SCALE`)."""
-    return round(SOURCE_STRENGTH["semantic"] / (1.0 + max(rank, 0) / SEMANTIC_RANK_SCALE), 6)
+    return round(SOURCE_STRENGTH["semantic"] / (1.0 + max(rank, 0) / float(scale)), 6)
 
 
 def _lexical_anchors(

{bgts_context_engine-0.2.2 → bgts_context_engine-0.2.3}/src/bce/core/orchestrator/orchestrator.py

@@ -25,6 +25,7 @@ from dataclasses import dataclass, field
 
 from bce.core.orchestrator.anchors import AnchorResult
 from bce.core.orchestrator.expand import expand_from_anchors, to_candidates
+from bce.core.orchestrator.profile import RetrievalProfile
 from bce.core.orchestrator.text import query_terms
 from bce.core.scoring.engine import (
     CONTAINER_KINDS,
@@ -59,6 +60,8 @@ PER_CONTAINER_SHARE = 0.3
 #: that split is where the model is weak (recall 10 %); on the holdout split, where the model is
 #: stronger (18 %), 0.3 let three of the model's own hits slip from the top 5 / out of the list.
 #: 0.5 is the a-priori choice: half the list is the model's, whatever the engine thinks.
+#: This is the voyage-code-4 value; another embedding model may carry its own share and a
+#: guard on its top ranks in its :class:`~bce.core.orchestrator.profile.RetrievalProfile`.
 SEMANTIC_RESERVE_SHARE = 0.5
 #: Diversity rule 3: at most ceil(n * share) container symbols among the first n (a class is a
 #: location, not a change site; its members carry the change).
@@ -95,9 +98,12 @@ class RetrievalOrchestrator:
         self,
         repository: GraphRepository,
         weights: ScoreWeights | None = None,
+        profile: RetrievalProfile | None = None,
     ) -> None:
         self.repository = repository
         self.weights = weights or ScoreWeights()
+        #: Model-specific narrowing constants; ``None`` = the voyage defaults.
+        self.profile = profile
 
     def retrieve(
         self,
@@ -141,7 +147,7 @@ class RetrievalOrchestrator:
         timings["score_ms"] = _elapsed_ms(stage)
 
         stage = time.perf_counter()
-        narrowed = narrow(ranked, max_candidates)
+        narrowed = narrow(ranked, max_candidates, profile=self.profile)
         timings["narrow_ms"] = _elapsed_ms(stage)
 
         signals = {c.symbol_id: c.task_signal for c in ranked if c.task_signal > 0.0}
@@ -168,7 +174,9 @@ def _container_of(cand: Candidate) -> str | None:
     return "::".join(segments[:-1])
 
 
-def narrow(ranked: list[Candidate], max_candidates: int) -> list[Candidate]:
+def narrow(
+    ranked: list[Candidate], max_candidates: int, *, profile: RetrievalProfile | None = None
+) -> list[Candidate]:
     """Deterministic, diversity-aware top-N over an already ranked (score desc, id asc) list.
 
     The list is built one slot at a time from two streams - the embedding model's ranks (rank
@@ -176,9 +184,15 @@ def narrow(ranked: list[Candidate], max_candidates: int) -> list[Candidate]:
     ``narrow(pool, n) == narrow(pool, n + 1)[:n]``: asking for a longer list never changes what
     the shorter one was (K-monotonic), and a K=20 run can be cut to K=10 exactly.
 
-    Slot ``i`` (1-based) goes to:
+    ``profile`` supplies the two model-specific constants (default: the voyage values,
+    :data:`SEMANTIC_RESERVE_SHARE` and no guard). Slot ``i`` (1-based) goes to:
 
-    1. the model's next rank while fewer than ``i * SEMANTIC_RESERVE_SHARE`` slots came from the
+    0. the model's next rank while ``i <= profile.semantic_guard_ranks`` (the guard: the model's
+       first ``n`` ranks *are* slots 1..n, whatever the engine scored - only the container cap
+       can skip one). A model whose correct hits spread over ranks 1-15 (jina) needs this;
+       without it the engine's agreement with the model's *deeper* ranks satisfies the share
+       below and its own picks push the model's ranks 7-15 to positions 11-19 or out;
+    1. the model's next rank while fewer than ``i * semantic_reserve_share`` slots came from the
        model stream (so position 1 is always the model's #1, and the model's top ranks can never
        be ranked out of the answer);
     2. else the engine's head (score order). An engine pick that *is* one of the model's ranks
@@ -206,6 +220,8 @@ def narrow(ranked: list[Candidate], max_candidates: int) -> list[Candidate]:
     """
     if max_candidates <= 0 or not ranked:
         return []
+    reserve_share = SEMANTIC_RESERVE_SHARE if profile is None else profile.semantic_reserve_share
+    guard_ranks = 0 if profile is None else max(0, int(profile.semantic_guard_ranks))
 
     model_stream = sorted(
         (c for c in ranked if c.semantic_rank is not None),
@@ -261,7 +277,7 @@ def narrow(ranked: list[Candidate], max_candidates: int) -> list[Candidate]:
         if model_head is None and engine_head is None:
             break
         if model_head is not None and (
-            engine_head is None or model_taken < slot * SEMANTIC_RESERVE_SHARE
+            engine_head is None or slot <= guard_ranks or model_taken < slot * reserve_share
         ):
             take(model_head, from_model=True)
         elif engine_head is not None: