claude-smart 0.2.42 → 0.2.44

package/plugin/vendor/reflexio/reflexio/benchmarks/retrieval_latency/README.md

@@ -0,0 +1,109 @@
+# Retrieval Latency Benchmark
+
+> Part of the [Reflexio Code Map](../../README.md). Related to the server's [Unified Search Service](../../server/README.md#unified-search-service) and [Storage](../../server/README.md#storage) components.
+
+Measures end-to-end latency for profile search, user/agent playbook search,
+and unified cross-entity search across storage backends and corpus sizes.
+
+## What it measures
+
+Four retrieval types × two layers × N storage backends × K corpus sizes.
+
+**Layers:**
+
+| Layer      | What it times                                               |
+|------------|-------------------------------------------------------------|
+| `service`  | `reflexio.search_*()` called directly in-process            |
+| `http`     | FastAPI `TestClient` POST to `/api/search_*` (no TCP — ASGI transport only) |
+
+The `http` layer measures FastAPI routing, Pydantic serialization, and
+middleware cost on top of the service layer. It does *not* include kernel
+socket or network RTT — the goal is to isolate framework overhead from
+storage work. Add a uvicorn-backed variant later if you need wall-clock HTTP.
+
+**Retrieval types:** `profile`, `user_playbook`, `agent_playbook`, `unified`.
+
+**Backends:** `sqlite` (always available) and `supabase` (requires a local
+`supabase start` + `SUPABASE_ANON_KEY` / `SUPABASE_DB_URL` env vars; skipped
+gracefully otherwise).
+
+## Controlling embedder cost
+
+Query embeddings are pre-cached on disk at
+`~/.cache/reflexio-benchmarks/embeddings-<model>.json`. First run populates
+the cache via the real embedding API (requires `OPENAI_API_KEY` or the
+configured provider key); subsequent runs are offline.
+
+Document embeddings during corpus seeding use deterministic hash-derived
+fake vectors. Document vector *content* doesn't affect query-time latency
+(only row count does), so this is a sound optimization. Query-time
+retrieval always uses the real cached query vector.
+
+Query reformulation (a separate LLM call) is disabled on every request so
+it doesn't contaminate the timing — it's a known LLM cost, orthogonal to
+retrieval.
+
+## Usage
+
+```bash
+# Default sweep: sizes 100/1000/10000, sqlite + supabase if available,
+# both layers, all four retrieval types, 50 trials + 5 warmup per cell.
+uv run python -m reflexio.benchmarks.retrieval_latency.bench
+
+# Small smoke run (for local iteration):
+uv run python -m reflexio.benchmarks.retrieval_latency.bench \
+    --sizes 100 --trials 10 --warmup 2 \
+    --backend sqlite --layer service --retrieval profile
+
+# Diff against a committed baseline:
+uv run python -m reflexio.benchmarks.retrieval_latency.bench \
+    --baseline tests/benchmarks/baseline.json
+```
+
+## Output
+
+Each run writes `results.json` and `report.md` to
+`reflexio/benchmarks/retrieval_latency/results/<timestamp>/` — next to the
+script itself, so reports travel with the code that produced them (override
+with `--output-dir`). The `results/` directory is gitignored; commit a report
+manually only when you want to save it as a baseline or reference.
+
+`results.json` contains the full config block, per-cell aggregated stats,
+and every raw trial row. `report.md` renders the stats as markdown tables,
+one per retrieval type, rows grouped by `(backend, layer)`. Cell format:
+`p50 / p95 (mean)` in milliseconds. When `--baseline` is passed, a ΔP95
+column flags cells where p95 has grown by 20% or more (`⚠`).
+
+## Interpreting the numbers
+
+Sanity checks to run on any report:
+
+- p95 should scale sub-linearly with N (both FTS and vector indexes are built).
+- HTTP layer > service layer on the same cell (framework tax).
+- Unified search ≈ `max(profile, user_playbook, agent_playbook)` plus a small
+  Phase A fixed cost — unified runs the three entity searches in parallel.
+- If a cell has wildly higher variance than its neighbors, suspect GC or a
+  cold cache — re-run with more warmup.
+
+## Pytest smoke test
+
+`tests/benchmarks/test_retrieval_latency_smoke.py` runs a tiny version of
+this benchmark at `N=50, trials=10, sqlite + service only` and asserts that
+p95 has not regressed past 3× the committed baseline. It's marked
+`skip_in_precommit`, so it runs in `check-and-test` but not on every commit. To
+regenerate the baseline after an intentional perf change:
+
+```bash
+uv run python -m reflexio.benchmarks.retrieval_latency.bench \
+    --sizes 50 --trials 10 --warmup 2 \
+    --backend sqlite --layer service \
+    --output-dir tests/benchmarks/tmp/
+cp tests/benchmarks/tmp/results.json tests/benchmarks/baseline.json
+```
+
+Then commit the new `baseline.json`.
+
+## See Also
+
+- [Code Map (root README)](../../README.md) -- high-level overview of all Reflexio components
+- [Server README](../../server/README.md) -- backend architecture including search and storage

package/plugin/vendor/reflexio/reflexio/benchmarks/retrieval_latency/__init__.py

@@ -0,0 +1 @@
+"""Retrieval latency benchmark — profile, playbook, unified search."""

package/plugin/vendor/reflexio/reflexio/benchmarks/retrieval_latency/backends.py

@@ -0,0 +1,175 @@
+"""
+Backend setup helpers for the retrieval latency benchmark.
+
+Each supported backend is exposed as a context manager that:
+
+1. Builds a ``Config`` with the appropriate ``StorageConfig``.
+2. Instantiates a ``Reflexio`` facade and obtains the underlying storage.
+3. Yields a :class:`BackendHandle` (reflexio instance + storage + org_id).
+4. Tears down temporary resources on exit.
+
+SQLite is always available. Supabase is only available when the local
+stack is running (``supabase start``) and the expected env vars are set; in
+all other cases the helper yields ``None`` so the benchmark loop can skip
+cleanly with a clear warning.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import tempfile
+from collections.abc import Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+
+from reflexio.lib.reflexio_lib import Reflexio
+from reflexio.models.config_schema import (
+    Config,
+    StorageConfigSQLite,
+    StorageConfigSupabase,
+)
+from reflexio.server.services.configurator.configurator import DefaultConfigurator
+from reflexio.server.services.storage.storage_base import BaseStorage
+
+logger = logging.getLogger(__name__)
+
+BENCH_ORG_ID = "bench_org"
+
+
+@dataclass
+class BackendHandle:
+    """
+    Handle to a live Reflexio facade wired to a specific storage backend.
+
+    Attributes:
+        name (str): Short backend identifier, e.g. ``"sqlite"``.
+        reflexio (Reflexio): Service-layer facade — call ``search_user_profiles``
+            etc. directly on this for the service layer benchmark.
+        storage (BaseStorage): Underlying storage instance, needed for
+            swapping ``_get_embedding`` during seeding and the timed loop.
+        org_id (str): Organization ID used for this benchmark run.
+    """
+
+    name: str
+    reflexio: Reflexio
+    storage: BaseStorage
+    org_id: str
+
+
+def _make_reflexio(config: Config, org_id: str) -> Reflexio:
+    """
+    Build a Reflexio instance from an in-memory ``Config``.
+
+    Args:
+        config (Config): Fully populated benchmark config.
+        org_id (str): Organization ID for the Reflexio instance.
+
+    Returns:
+        Reflexio: A facade with storage wired up through the configurator.
+    """
+    configurator = DefaultConfigurator(org_id=org_id, config=config)
+    return Reflexio(org_id=org_id, configurator=configurator)
+
+
+@contextmanager
+def sqlite_backend() -> Iterator[BackendHandle]:
+    """
+    Construct an isolated SQLite-backed Reflexio instance in a tmpdir.
+
+    Mirrors the contract-test storage fixture pattern: fresh directory,
+    fresh database, no shared state with other runs.
+
+    Yields:
+        BackendHandle: A live handle with name ``"sqlite"``.
+    """
+    with tempfile.TemporaryDirectory(prefix="reflexio-bench-sqlite-") as tmp:
+        db_path = str(Path(tmp) / "reflexio.db")
+        config = Config(storage_config=StorageConfigSQLite(db_path=db_path))
+        reflexio = _make_reflexio(config, BENCH_ORG_ID)
+        storage = reflexio._get_storage()
+        yield BackendHandle(
+            name="sqlite",
+            reflexio=reflexio,
+            storage=storage,
+            org_id=BENCH_ORG_ID,
+        )
+
+
+def _supabase_env_ready() -> StorageConfigSupabase | None:
+    """
+    Read Supabase env vars and return a populated config, or ``None``.
+
+    Falls back to the default local ``supabase start`` URL when only the
+    keys are provided.
+
+    Returns:
+        StorageConfigSupabase | None: A config if all three fields can be
+        resolved, otherwise ``None``.
+    """
+    url = os.environ.get("SUPABASE_URL", "http://127.0.0.1:54321")
+    key = os.environ.get("SUPABASE_ANON_KEY") or os.environ.get("SUPABASE_KEY")
+    db_url = os.environ.get("SUPABASE_DB_URL")
+    if not (key and db_url):
+        return None
+    return StorageConfigSupabase(url=url, key=key, db_url=db_url)
+
+
+@contextmanager
+def supabase_backend() -> Iterator[BackendHandle | None]:
+    """
+    Construct a Supabase-backed Reflexio instance if the local stack is up.
+
+    Yields ``None`` (and logs a warning) when the Supabase env vars are
+    missing or the connection fails, so the benchmark loop can skip this
+    backend gracefully without aborting the whole run.
+
+    Yields:
+        BackendHandle | None: Live handle, or ``None`` if unavailable.
+    """
+    storage_config = _supabase_env_ready()
+    if storage_config is None:
+        logger.warning(
+            "Supabase backend skipped: set SUPABASE_ANON_KEY and SUPABASE_DB_URL "
+            "(and optionally SUPABASE_URL) to enable it."
+        )
+        yield None
+        return
+
+    config = Config(storage_config=storage_config)
+    try:
+        reflexio = _make_reflexio(config, BENCH_ORG_ID)
+        storage = reflexio._get_storage()
+    except Exception as err:  # noqa: BLE001
+        logger.warning(
+            "Supabase backend skipped: failed to connect — %s. "
+            "Is `supabase start` running?",
+            err,
+        )
+        yield None
+        return
+
+    try:
+        yield BackendHandle(
+            name="supabase",
+            reflexio=reflexio,
+            storage=storage,
+            org_id=BENCH_ORG_ID,
+        )
+    finally:
+        # Best-effort cleanup: wipe the benchmark org's rows so repeated
+        # runs don't accumulate forever. Swallow errors — a failing cleanup
+        # must not mask a real measurement failure.
+        try:
+            reflexio.delete_all_profiles_bulk()
+            reflexio.delete_all_user_playbooks_bulk()
+            reflexio.delete_all_agent_playbooks_bulk()
+        except Exception as err:  # noqa: BLE001
+            logger.warning("Supabase cleanup failed: %s", err)
+
+
+BACKENDS = {
+    "sqlite": sqlite_backend,
+    "supabase": supabase_backend,
+}