clef-compactor 0.2.0__tar.gz → 0.2.1__tar.gz

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/.github/workflows/publish.yml

@@ -17,4 +17,4 @@ jobs:
         with:
           python-version: "3.12"
       - run: uv build
-      - run: uv publish
+      - run: uv publish --check-url https://pypi.org/simple

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/.gitignore

@@ -39,3 +39,5 @@ Thumbs.db
 .env
 .env.*
 *.pem
+kaggle-out/
+kaggle-out/

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: clef-compactor
-Version: 0.2.0
+Version: 0.2.1
 Summary: Query-aware RAG context compaction using Cloudflare's Clef. Keep the evidence, cut the noise.
 Project-URL: Homepage, https://github.com/Gjusev/clef-compactor
 Project-URL: Repository, https://github.com/Gjusev/clef-compactor
@@ -194,21 +194,38 @@ query_engine = RetrieverQueryEngine(
 
 ## Measured results
 
-Replay run (deterministic simulated scorer, 20 cases, 104 chunks, seed
-20261001, reproducible with `python evals/run_eval.py`). This validates the
-pipeline, not model quality; see [Limitations](#limitations).
+Three measurement sources, each labelled for what it actually proves.
+
+**Real model, modest hardware** — open-weights `Cloudflare/clef-flash` (9B),
+float16 sharded across 2x Kaggle T4, 20 cases, 104 chunks. Produced by
+`python evals/run_eval.py --mode local`; results committed in
+`evals/results/results-local-t4x2.json` and reproducible from the
+[`clef-compactor-evals`](https://www.kaggle.com/code/gjusev/clef-compactor-evals)
+kernel.
 
 | metric | value |
 |---|---|
-| chunk accuracy | 0.990 |
-| kept precision | 1.000 |
-| relevant recall | 0.984 |
-| kept F1 | 0.992 |
-| context tokens saved | 34.0% |
-| cost per 1k calls | $0.019 (input tokens at $0.24/M) |
-
-Live-API numbers are pending credentials and will replace these once a run
-lands. The harness is ready: `python evals/run_eval.py --mode live`.
+| chunk accuracy | 0.712 |
+| kept precision | 0.771 |
+| relevant recall | 0.746 |
+| kept F1 | 0.758 |
+| context tokens saved | 38.4% |
+| scoring latency p50 / p95 | 1,274 / 1,407 ms |
+| cost per 1k calls | $0.00 (self-hosted weights) |
+
+**Pipeline validation** — deterministic simulated scorer, same dataset, seed
+20261001 (`python evals/run_eval.py`). Proves the ranking and budget
+machinery, not model quality: chunk accuracy 0.990, kept F1 0.992, 34.0%
+tokens saved.
+
+**Hosted API** — pending credentials; `--mode live` produces it.
+
+Reading the real numbers plainly: on a T4 the 9B model makes the right
+keep/drop call 71% of the time, keeps three quarters of the relevant chunks,
+and still removes 38% of the context tokens. Local latency is dominated by
+the modest GPU and the torch fallback for Qwen3.5's linear attention (the
+fast-path kernels were not installed in the kernel); the hosted endpoint
+reports 38.8 ms median for clef-flash on Cloudflare's own hardware.
 
 ## clef vs laya
 
@@ -248,9 +265,10 @@ and honor `Retry-After`.
 
 ## Limitations
 
-- **The committed accuracy numbers come from a simulated scorer.** They prove
-  the pipeline ranks and cuts correctly under noisy probabilities. They do not
-  measure Clef. Run `--mode live` before quoting quality.
+- **Real-model numbers are from a 9B model on a T4 pair, not from the hosted
+  endpoint.** The hosted API (and the 27B model, which needs ~54 GB) may
+  score better. Measuring the hosted endpoint is one command away:
+  `--mode live` with credentials.
 - **Latency.** clef's median decision latency is 209 ms (38.8 ms for
   clef-flash) plus network. laya keeps the whole job local at 5.8 ms. If you
   need sub-10 ms compaction on every request, see

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/README.md

@@ -153,21 +153,38 @@ query_engine = RetrieverQueryEngine(
 
 ## Measured results
 
-Replay run (deterministic simulated scorer, 20 cases, 104 chunks, seed
-20261001, reproducible with `python evals/run_eval.py`). This validates the
-pipeline, not model quality; see [Limitations](#limitations).
+Three measurement sources, each labelled for what it actually proves.
+
+**Real model, modest hardware** — open-weights `Cloudflare/clef-flash` (9B),
+float16 sharded across 2x Kaggle T4, 20 cases, 104 chunks. Produced by
+`python evals/run_eval.py --mode local`; results committed in
+`evals/results/results-local-t4x2.json` and reproducible from the
+[`clef-compactor-evals`](https://www.kaggle.com/code/gjusev/clef-compactor-evals)
+kernel.
 
 | metric | value |
 |---|---|
-| chunk accuracy | 0.990 |
-| kept precision | 1.000 |
-| relevant recall | 0.984 |
-| kept F1 | 0.992 |
-| context tokens saved | 34.0% |
-| cost per 1k calls | $0.019 (input tokens at $0.24/M) |
-
-Live-API numbers are pending credentials and will replace these once a run
-lands. The harness is ready: `python evals/run_eval.py --mode live`.
+| chunk accuracy | 0.712 |
+| kept precision | 0.771 |
+| relevant recall | 0.746 |
+| kept F1 | 0.758 |
+| context tokens saved | 38.4% |
+| scoring latency p50 / p95 | 1,274 / 1,407 ms |
+| cost per 1k calls | $0.00 (self-hosted weights) |
+
+**Pipeline validation** — deterministic simulated scorer, same dataset, seed
+20261001 (`python evals/run_eval.py`). Proves the ranking and budget
+machinery, not model quality: chunk accuracy 0.990, kept F1 0.992, 34.0%
+tokens saved.
+
+**Hosted API** — pending credentials; `--mode live` produces it.
+
+Reading the real numbers plainly: on a T4 the 9B model makes the right
+keep/drop call 71% of the time, keeps three quarters of the relevant chunks,
+and still removes 38% of the context tokens. Local latency is dominated by
+the modest GPU and the torch fallback for Qwen3.5's linear attention (the
+fast-path kernels were not installed in the kernel); the hosted endpoint
+reports 38.8 ms median for clef-flash on Cloudflare's own hardware.
 
 ## clef vs laya
 
@@ -207,9 +224,10 @@ and honor `Retry-After`.
 
 ## Limitations
 
-- **The committed accuracy numbers come from a simulated scorer.** They prove
-  the pipeline ranks and cuts correctly under noisy probabilities. They do not
-  measure Clef. Run `--mode live` before quoting quality.
+- **Real-model numbers are from a 9B model on a T4 pair, not from the hosted
+  endpoint.** The hosted API (and the 27B model, which needs ~54 GB) may
+  score better. Measuring the hosted endpoint is one command away:
+  `--mode live` with credentials.
 - **Latency.** clef's median decision latency is 209 ms (38.8 ms for
   clef-flash) plus network. laya keeps the whole job local at 5.8 ms. If you
   need sub-10 ms compaction on every request, see

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/docs/index.html

@@ -164,16 +164,17 @@
 
 <section id="results" class="wrap">
   <div class="kicker">measured results</div>
-  <h2 class="serif">Replay baseline, committed and reproducible</h2>
-  <p class="lede">Deterministic simulated scorer, 20 cases, 104 chunks, seed 20261001. This validates the pipeline; it does not measure Clef. Live numbers land via <span class="mono">evals/run_eval.py --mode live</span>.</p>
+  <h2 class="serif">Real model, measured on real hardware</h2>
+  <p class="lede">Open-weights clef-flash 9B, float16 on 2x Kaggle T4, 20 hand-labelled cases, 104 chunks. Reproducible from the <a href="https://www.kaggle.com/code/gjusev/clef-compactor-evals">public kernel</a>. The replay baseline (simulated scorer) and a hosted-API run are documented in the repo.</p>
   <table class="reveal" style="--i:1">
-    <tr><th>metric</th><th>value</th></tr>
-    <tr><td>chunk accuracy</td><td class="num">0.990</td></tr>
-    <tr><td>kept precision</td><td class="num">1.000</td></tr>
-    <tr><td>relevant recall</td><td class="num">0.984</td></tr>
-    <tr><td>kept F1</td><td class="num">0.992</td></tr>
-    <tr><td>context tokens saved</td><td class="num best">34.0%</td></tr>
-    <tr><td>cost per 1k calls</td><td class="num">$0.019</td></tr>
+    <tr><th>metric</th><th>open-weights 9B on 2xT4</th><th>replay baseline</th></tr>
+    <tr><td>chunk accuracy</td><td class="num">0.712</td><td class="num">0.990</td></tr>
+    <tr><td>kept precision</td><td class="num">0.771</td><td class="num">1.000</td></tr>
+    <tr><td>relevant recall</td><td class="num">0.746</td><td class="num">0.984</td></tr>
+    <tr><td>kept F1</td><td class="num">0.758</td><td class="num">0.992</td></tr>
+    <tr><td>context tokens saved</td><td class="num best">38.4%</td><td class="num">34.0%</td></tr>
+    <tr><td>scoring latency p50</td><td class="num">1,274 ms</td><td class="num">&lt;1 ms</td></tr>
+    <tr><td>cost per 1k calls</td><td class="num best">$0.00 self-hosted</td><td class="num">$0.019</td></tr>
   </table>
 </section>
 

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/evals/README.md

@@ -16,7 +16,10 @@ by hand against the query; they are the ground truth for every metric.
 # Pipeline validation: real compaction code, deterministic simulated scorer
 python evals/run_eval.py
 
-# Real quality numbers: needs CLEF_ACCOUNT_ID and CLEF_API_TOKEN
+# Real model weights, no Cloudflare account: 9B on a local GPU (T4 pair tested)
+python evals/run_eval.py --mode local --model-path Cloudflare/clef-flash
+
+# Hosted endpoint: the published prices and latencies apply
 python evals/run_eval.py --mode live
 
 # CI gate: exit code 1 when chunk accuracy drops below the threshold
@@ -26,6 +29,16 @@ python evals/run_eval.py --min-accuracy 0.9
 python evals/run_eval.py --compare-laya
 ```
 
+## Measured so far
+
+`results/results.json` is the replay baseline (simulated scorer).
+`results/results-local-t4x2.json` is the real open-weights clef-flash 9B
+measured on a Kaggle T4 pair via the
+[`clef-compactor-evals`](https://www.kaggle.com/code/gjusev/clef-compactor-evals)
+kernel: chunk accuracy 0.712, kept F1 0.758, 38.4% of context tokens removed,
+scoring latency ~1.27 s p50 (T4, torch attention fallback). The 27B model
+needs ~54 GB and does not fit on that hardware.
+
 Every run writes `results/results.json`: aggregate metrics plus one entry per
 case. The committed file was produced by `--mode replay`, so it is
 reproducible byte for byte given the same seed.

clef_compactor-0.2.1/evals/backends.py

@@ -0,0 +1,159 @@
+"""Local inference backend: run the open-weights Clef models without the API.
+
+Wraps ``joint_schema_model.systemone`` from the Hugging Face release
+(https://huggingface.co/Cloudflare/clef) in the same ``ask()`` interface the
+sync compactor expects, so the whole eval pipeline runs unmodified against
+weights loaded on a local GPU.
+
+The stock ``load_release_model`` pins the whole backbone to one device via
+``device_map={"": device}``. That fits clef-flash (9B, ~18 GB in fp16) on a
+single H200 but not on a single 16 GB T4, so :class:`LocalClefClient`
+replicates the loader with ``device_map="auto"`` and places the joint schema
+head on the device that holds the backbone's final layers.
+
+Requirements (not package dependencies; installed by the Kaggle kernel):
+``torch``, ``transformers>=5.10``, ``accelerate``, ``safetensors``,
+``huggingface_hub``, ``pillow``.
+"""
+
+from __future__ import annotations
+
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from clef_compactor.config import Settings
+from clef_compactor.models import ClefReply, TokenUsage
+
+__all__ = ["LocalClefClient"]
+
+
+class LocalClefClient:
+    """Drop-in ``ask()`` client backed by locally loaded Clef weights.
+
+    Args:
+        settings: Validated settings; ``model`` labels the replies.
+        model_path: HF repo id (e.g. ``"Cloudflare/clef-flash"``) or a local
+            snapshot directory.
+        dtype: Torch dtype name for the backbone, ``"float16"`` for Turing GPUs
+            (T4), ``"bfloat16"`` on Ampere or newer.
+        device_map: Accelerate device map, ``"auto"`` to shard across GPUs.
+        max_memory_gib: Per-GPU memory ceiling used with ``device_map="auto"``.
+        max_length: Token bound forwarded to ``encode_record`` (default 16384).
+    """
+
+    def __init__(
+        self,
+        settings: Settings,
+        model_path: str,
+        *,
+        dtype: str = "float16",
+        device_map: str = "auto",
+        max_memory_gib: int = 15,
+        max_length: int = 16384,
+    ) -> None:
+        import torch  # heavy: imported lazily so import cost stays off the API path
+
+        self.torch = torch
+        self.settings = settings
+        self.max_length = max_length
+
+        path = Path(model_path)
+        if not path.is_dir():
+            from huggingface_hub import snapshot_download
+
+            path = Path(snapshot_download(model_path))
+
+        joint_schema_dir = str(path)
+        if joint_schema_dir not in sys.path:
+            sys.path.insert(0, joint_schema_dir)
+        import json
+
+        from joint_schema_model import ClefModel, JointSchemaHead  # type: ignore[import-not-found]
+        from safetensors.torch import load_file
+        from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
+
+        torch_dtype = getattr(torch, dtype)
+        backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
+            path,
+            dtype=torch_dtype,
+            device_map=device_map,
+            max_memory={index: f"{max_memory_gib}GiB" for index in range(torch.cuda.device_count())},
+        )
+        backbone.config.use_cache = False
+
+        head_config = json.loads((path / "joint_head_config.json").read_text())
+        head = JointSchemaHead(**head_config)
+        head.load_state_dict(load_file(path / "joint_head.safetensors"), strict=True)
+        head = head.to(device=self._last_device(backbone), dtype=torch_dtype)
+
+        self.model = ClefModel(backbone, head).eval()
+        self.processor = AutoProcessor.from_pretrained(path)
+
+    @staticmethod
+    def _last_device(backbone: Any) -> Any:
+        """The device holding the backbone's final layers (head must match).
+
+        ``hf_device_map`` values may be ints (``0``, ``1``) or strings
+        (``"cuda:1"``); normalise both.
+        """
+        import torch
+
+        devices = list(getattr(backbone, "hf_device_map", {}).values())
+        if not devices:
+            return torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        indexes = [int(str(value).split(":")[-1]) for value in devices]
+        return torch.device("cuda", max(indexes))
+
+    def ask(
+        self,
+        state: str | dict[str, Any],
+        questions: dict[str, dict[str, Any]],
+        *,
+        images: list[str] | None = None,
+        model: str | None = None,
+    ) -> ClefReply:
+        """Answer one scoring request with a single local forward pass."""
+        from joint_schema_model import systemone  # type: ignore[import-not-found]
+
+        if not questions:
+            raise ValueError("questions must contain at least one entry")
+        request = {
+            "model": model or self.settings.model,
+            "state": state,
+            "questions": questions,
+        }
+        started = time.perf_counter()
+        response = systemone(self._collate_on_head_device(), self.processor, request, max_length=self.max_length)
+        latency_ms = (time.perf_counter() - started) * 1000
+        return ClefReply(
+            model=str(response["model"]),
+            answers=response["answers"],
+            usage=TokenUsage.from_api(response.get("usage")),
+            request_id="local-weights",
+            latency_ms=latency_ms,
+        )
+
+    def _collate_on_head_device(self) -> Any:
+        """View of the model whose first parameter lives where the head does.
+
+        ``systemone`` collates the batch onto ``next(model.parameters()).device``.
+        Across a sharded load, the joint head reads ``token_ids`` from the batch
+        and indexes tensors on its own device, so the batch must be collated on
+        the head's device; accelerate's hooks move the backbone inputs to the
+        first shard as usual.
+        """
+        inner = self.model
+
+        class _CollateOnHeadDevice:
+            def parameters(self, *args: Any, **kwargs: Any) -> Any:
+                return inner.head.parameters()
+
+            def __call__(self, batch: dict[str, Any]) -> Any:
+                return inner(batch)
+
+            def __getattr__(self, name: str) -> Any:
+                return getattr(inner, name)
+
+        return _CollateOnHeadDevice()

clef_compactor-0.2.1/evals/results/results-local-t4x2.json

@@ -0,0 +1,274 @@
+{
+  "meta": {
+    "mode": "local",
+    "scorer": "open-weights Cloudflare/clef-flash (float16, local GPU)",
+    "transport": "local torch forward pass",
+    "dataset": [
+      "/kaggle/working/clef-compactor/evals/data/compaction_suite.jsonl"
+    ],
+    "relevance_threshold": 0.5,
+    "cost_model": "self-hosted open weights: no per-token API cost",
+    "price_note": "Output-token pricing is unpublished; only input tokens are ever costed."
+  },
+  "metrics": {
+    "cases": 20,
+    "chunks": 104,
+    "chunk_accuracy": 0.7115,
+    "kept_precision": 0.7705,
+    "relevant_recall": 0.746,
+    "kept_f1": 0.7581,
+    "input_tokens": 1582,
+    "output_tokens": 975,
+    "avg_saved_fraction": 0.3837,
+    "latency_ms": {
+      "p50": 1273.86,
+      "p95": 1406.64,
+      "p99": 3404.95
+    },
+    "api_input_tokens": 16229,
+    "cost_usd_per_1k_calls": 0.194748,
+    "cost_model": "self-hosted open weights: no per-token API cost"
+  },
+  "per_case": [
+    {
+      "case_id": "support-refund-001",
+      "correct": 4,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 74,
+      "output_tokens": 58,
+      "latency_ms": 3404.9475499999744,
+      "api_input_tokens": 778
+    },
+    {
+      "case_id": "support-shipping-002",
+      "correct": 4,
+      "total": 5,
+      "kept_relevant": 2,
+      "total_relevant": 2,
+      "kept_irrelevant": 1,
+      "input_tokens": 67,
+      "output_tokens": 45,
+      "latency_ms": 1406.6402149999249,
+      "api_input_tokens": 771
+    },
+    {
+      "case_id": "tech-python-003",
+      "correct": 5,
+      "total": 6,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 101,
+      "output_tokens": 76,
+      "latency_ms": 1362.4956160000465,
+      "api_input_tokens": 935
+    },
+    {
+      "case_id": "tech-db-004",
+      "correct": 2,
+      "total": 5,
+      "kept_relevant": 1,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 73,
+      "output_tokens": 32,
+      "latency_ms": 1252.008700000033,
+      "api_input_tokens": 783
+    },
+    {
+      "case_id": "finance-fees-005",
+      "correct": 5,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 0,
+      "input_tokens": 68,
+      "output_tokens": 44,
+      "latency_ms": 1249.5986689999654,
+      "api_input_tokens": 779
+    },
+    {
+      "case_id": "travel-visa-006",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 2,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 81,
+      "output_tokens": 57,
+      "latency_ms": 1261.9110020000335,
+      "api_input_tokens": 791
+    },
+    {
+      "case_id": "health-insurance-007",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 2,
+      "input_tokens": 87,
+      "output_tokens": 87,
+      "latency_ms": 1253.8356179999255,
+      "api_input_tokens": 795
+    },
+    {
+      "case_id": "devops-k8s-008",
+      "correct": 4,
+      "total": 5,
+      "kept_relevant": 2,
+      "total_relevant": 3,
+      "kept_irrelevant": 0,
+      "input_tokens": 78,
+      "output_tokens": 33,
+      "latency_ms": 831.2345519999553,
+      "api_input_tokens": 783
+    },
+    {
+      "case_id": "legal-privacy-009",
+      "correct": 4,
+      "total": 6,
+      "kept_relevant": 2,
+      "total_relevant": 4,
+      "kept_irrelevant": 0,
+      "input_tokens": 90,
+      "output_tokens": 26,
+      "latency_ms": 1363.2799339999337,
+      "api_input_tokens": 924
+    },
+    {
+      "case_id": "product-pricing-010",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 2,
+      "input_tokens": 69,
+      "output_tokens": 69,
+      "latency_ms": 1254.592740000021,
+      "api_input_tokens": 777
+    },
+    {
+      "case_id": "science-space-011",
+      "correct": 4,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 80,
+      "output_tokens": 66,
+      "latency_ms": 1273.8597500000424,
+      "api_input_tokens": 781
+    },
+    {
+      "case_id": "education-math-012",
+      "correct": 2,
+      "total": 5,
+      "kept_relevant": 1,
+      "total_relevant": 4,
+      "kept_irrelevant": 0,
+      "input_tokens": 103,
+      "output_tokens": 17,
+      "latency_ms": 1268.813750999925,
+      "api_input_tokens": 808
+    },
+    {
+      "case_id": "support-account-013",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 1,
+      "total_relevant": 3,
+      "kept_irrelevant": 0,
+      "input_tokens": 79,
+      "output_tokens": 21,
+      "latency_ms": 1265.0558409999348,
+      "api_input_tokens": 787
+    },
+    {
+      "case_id": "ops-oncall-014",
+      "correct": 6,
+      "total": 6,
+      "kept_relevant": 4,
+      "total_relevant": 4,
+      "kept_irrelevant": 0,
+      "input_tokens": 79,
+      "output_tokens": 57,
+      "latency_ms": 1367.6004610000518,
+      "api_input_tokens": 914
+    },
+    {
+      "case_id": "hr-leave-015",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 2,
+      "input_tokens": 76,
+      "output_tokens": 76,
+      "latency_ms": 1306.5989709999712,
+      "api_input_tokens": 784
+    },
+    {
+      "case_id": "ml-training-016",
+      "correct": 5,
+      "total": 6,
+      "kept_relevant": 3,
+      "total_relevant": 4,
+      "kept_irrelevant": 0,
+      "input_tokens": 83,
+      "output_tokens": 42,
+      "latency_ms": 1375.9791559999712,
+      "api_input_tokens": 918
+    },
+    {
+      "case_id": "ecommerce-inventory-017",
+      "correct": 5,
+      "total": 5,
+      "kept_relevant": 3,
+      "total_relevant": 3,
+      "kept_irrelevant": 0,
+      "input_tokens": 70,
+      "output_tokens": 49,
+      "latency_ms": 834.4232719999809,
+      "api_input_tokens": 778
+    },
+    {
+      "case_id": "security-auth-018",
+      "correct": 3,
+      "total": 5,
+      "kept_relevant": 2,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 67,
+      "output_tokens": 41,
+      "latency_ms": 1267.2506279999425,
+      "api_input_tokens": 772
+    },
+    {
+      "case_id": "nutrition-diet-019",
+      "correct": 4,
+      "total": 5,
+      "kept_relevant": 2,
+      "total_relevant": 3,
+      "kept_irrelevant": 0,
+      "input_tokens": 81,
+      "output_tokens": 44,
+      "latency_ms": 1292.3661680000578,
+      "api_input_tokens": 785
+    },
+    {
+      "case_id": "tools-git-020",
+      "correct": 2,
+      "total": 5,
+      "kept_relevant": 1,
+      "total_relevant": 3,
+      "kept_irrelevant": 1,
+      "input_tokens": 76,
+      "output_tokens": 35,
+      "latency_ms": 1304.9001580000095,
+      "api_input_tokens": 786
+    }
+  ]
+}

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/evals/results/results.json

@@ -7,7 +7,8 @@
       "C:\\Code Main\\clef-compactor\\evals\\data\\compaction_suite.jsonl"
     ],
     "relevance_threshold": 0.5,
-    "price_note": "Clef input tokens cost $0.24/M; output pricing is unpublished."
+    "cost_model": "input tokens x $0.24/M (Cloudflare published price)",
+    "price_note": "Output-token pricing is unpublished; only input tokens are ever costed."
   },
   "metrics": {
     "cases": 20,
@@ -20,9 +21,9 @@
     "output_tokens": 1044,
     "avg_saved_fraction": 0.3401,
     "latency_ms": {
-      "p50": 0.11,
-      "p95": 0.16,
-      "p99": 150.94
+      "p50": 0.12,
+      "p95": 0.17,
+      "p99": 156.81
     },
     "api_input_tokens": null,
     "cost_usd_per_1k_calls": 0.018984,
@@ -38,7 +39,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 74,
       "output_tokens": 43,
-      "latency_ms": 150.935500000287,
+      "latency_ms": 156.8135000006805,
       "api_input_tokens": null
     },
     {
@@ -50,7 +51,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 67,
       "output_tokens": 29,
-      "latency_ms": 0.1609999999345746,
+      "latency_ms": 0.16309999955410603,
       "api_input_tokens": null
     },
     {
@@ -62,7 +63,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 101,
       "output_tokens": 64,
-      "latency_ms": 0.15769999981785077,
+      "latency_ms": 0.1578999999765074,
       "api_input_tokens": null
     },
     {
@@ -74,7 +75,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 73,
       "output_tokens": 48,
-      "latency_ms": 0.1133000000663742,
+      "latency_ms": 0.1140000003942987,
       "api_input_tokens": null
     },
     {
@@ -86,7 +87,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 68,
       "output_tokens": 44,
-      "latency_ms": 0.10729999985414906,
+      "latency_ms": 0.10680000013962854,
       "api_input_tokens": null
     },
     {
@@ -98,7 +99,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 81,
       "output_tokens": 57,
-      "latency_ms": 0.11080000012952951,
+      "latency_ms": 0.1368000002912595,
       "api_input_tokens": null
     },
     {
@@ -110,7 +111,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 87,
       "output_tokens": 62,
-      "latency_ms": 0.11819999963336159,
+      "latency_ms": 0.17209999987244373,
       "api_input_tokens": null
     },
     {
@@ -122,7 +123,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 78,
       "output_tokens": 53,
-      "latency_ms": 0.11480000011943048,
+      "latency_ms": 0.13159999980416615,
       "api_input_tokens": null
     },
     {
@@ -134,7 +135,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 90,
       "output_tokens": 56,
-      "latency_ms": 0.12699999979304266,
+      "latency_ms": 0.13320000016392441,
       "api_input_tokens": null
     },
     {
@@ -146,7 +147,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 69,
       "output_tokens": 45,
-      "latency_ms": 0.10059999976874678,
+      "latency_ms": 0.11009999980160501,
       "api_input_tokens": null
     },
     {
@@ -158,7 +159,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 80,
       "output_tokens": 52,
-      "latency_ms": 0.10849999989659409,
+      "latency_ms": 0.11050000011891825,
       "api_input_tokens": null
     },
     {
@@ -170,7 +171,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 103,
       "output_tokens": 90,
-      "latency_ms": 0.12549999973998638,
+      "latency_ms": 0.12269999933778308,
       "api_input_tokens": null
     },
     {
@@ -182,7 +183,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 79,
       "output_tokens": 53,
-      "latency_ms": 0.10330000031899544,
+      "latency_ms": 0.10440000005473848,
       "api_input_tokens": null
     },
     {
@@ -194,7 +195,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 79,
       "output_tokens": 57,
-      "latency_ms": 0.11709999989761855,
+      "latency_ms": 0.11859999995067483,
       "api_input_tokens": null
     },
     {
@@ -206,7 +207,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 76,
       "output_tokens": 40,
-      "latency_ms": 0.10100000008606003,
+      "latency_ms": 0.12630000037461286,
       "api_input_tokens": null
     },
     {
@@ -218,7 +219,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 83,
       "output_tokens": 58,
-      "latency_ms": 0.12579999975059764,
+      "latency_ms": 0.1384999995934777,
       "api_input_tokens": null
     },
     {
@@ -230,7 +231,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 70,
       "output_tokens": 49,
-      "latency_ms": 0.10109999993801466,
+      "latency_ms": 0.1022000005832524,
       "api_input_tokens": null
     },
     {
@@ -242,7 +243,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 67,
       "output_tokens": 39,
-      "latency_ms": 0.09580000005371403,
+      "latency_ms": 0.09659999977884581,
       "api_input_tokens": null
     },
     {
@@ -254,7 +255,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 81,
       "output_tokens": 54,
-      "latency_ms": 0.10539999993852689,
+      "latency_ms": 0.10689999999158317,
       "api_input_tokens": null
     },
     {
@@ -266,7 +267,7 @@
       "kept_irrelevant": 0,
       "input_tokens": 76,
       "output_tokens": 51,
-      "latency_ms": 0.10370000018156134,
+      "latency_ms": 0.10449999990669312,
       "api_input_tokens": null
     }
   ]

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/evals/run_eval.py

@@ -181,7 +181,7 @@ def run_case(engine: ClefCompactor, case: EvalCase) -> CaseOutcome:
     )
 
 
-def aggregate(outcomes: Sequence[CaseOutcome]) -> dict[str, Any]:
+def aggregate(outcomes: Sequence[CaseOutcome], cost_model: str) -> dict[str, Any]:
     """Reduce per-case outcomes into the reported metric block."""
     total_chunks = sum(outcome.total for outcome in outcomes)
     correct = sum(outcome.correct for outcome in outcomes)
@@ -219,7 +219,7 @@ def aggregate(outcomes: Sequence[CaseOutcome]) -> dict[str, Any]:
         "latency_ms": {"p50": percentile_share(0.50), "p95": percentile_share(0.95), "p99": percentile_share(0.99)},
         "api_input_tokens": api_input_tokens or None,
         "cost_usd_per_1k_calls": round(cost_per_1k, 6),
-        "cost_model": f"input tokens x ${INPUT_PRICE_PER_M}/M (Cloudflare published price)",
+        "cost_model": cost_model,
     }
 
 
@@ -239,7 +239,12 @@ def print_laya_comparison() -> None:
 def main(argv: Sequence[str] | None = None) -> int:
     """Entry point; returns a process exit code (1 below --min-accuracy)."""
     parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    parser.add_argument("--mode", choices=("replay", "live"), default="replay")
+    parser.add_argument("--mode", choices=("replay", "live", "local"), default="replay")
+    parser.add_argument("--model-path", default="Cloudflare/clef-flash",
+                        help="HF repo id or snapshot dir for --mode local")
+    parser.add_argument("--dtype", default="float16", choices=("float16", "bfloat16", "float32"),
+                        help="Torch dtype for --mode local (float16 on T4)")
+    parser.add_argument("--device-map", default="auto", help="Accelerate device map for --mode local")
     parser.add_argument("--data", default=DATA_GLOB, help="Glob of dataset .jsonl files")
     parser.add_argument("--out", default=str(DEFAULT_OUT), help="Path of the results JSON")
     parser.add_argument("--seed", type=int, default=20261001, help="Replay noise seed")
@@ -253,6 +258,28 @@ def main(argv: Sequence[str] | None = None) -> int:
         engine = ClefCompactor()
         scorer = "cloudflare-clef (live)"
         transport = "https://api.cloudflare.com"
+        cost_model = "input tokens x $0.24/M (Cloudflare published price)"
+    elif args.mode == "local":
+        sys.path.insert(0, str(Path(__file__).resolve().parent))
+        from backends import LocalClefClient
+
+        model_label = args.model_path.split("/")[-1]
+        client = LocalClefClient(
+            Settings(account_id="local", api_token="local", model=model_label, max_retries=0),
+            args.model_path,
+            dtype=args.dtype,
+            device_map=args.device_map,
+        )
+        engine = ClefCompactor(
+            account_id="local",
+            api_token="local",
+            model=model_label,
+            relevance_threshold=args.threshold,
+            client=client,  # type: ignore[arg-type]
+        )
+        scorer = f"open-weights {args.model_path} ({args.dtype}, local GPU)"
+        transport = "local torch forward pass"
+        cost_model = "self-hosted open weights: no per-token API cost"
     else:
         labels_by_query = {case.query: case.labels for case in cases}
         engine = ClefCompactor(
@@ -263,9 +290,10 @@ def main(argv: Sequence[str] | None = None) -> int:
         )
         scorer = f"simulated-clef seed={args.seed} (pipeline validation, not model quality)"
         transport = "in-process mock"
+        cost_model = "input tokens x $0.24/M (Cloudflare published price)"
 
     outcomes = [run_case(engine, case) for case in cases]
-    metrics = aggregate(outcomes)
+    metrics = aggregate(outcomes, cost_model)
     payload = {
         "meta": {
             "mode": args.mode,
@@ -273,7 +301,8 @@ def main(argv: Sequence[str] | None = None) -> int:
             "transport": transport,
             "dataset": sorted(glob_module.glob(args.data)),
             "relevance_threshold": args.threshold,
-            "price_note": "Clef input tokens cost $0.24/M; output pricing is unpublished.",
+            "cost_model": cost_model,
+            "price_note": "Output-token pricing is unpublished; only input tokens are ever costed.",
         },
         "metrics": metrics,
         "per_case": [vars(outcome) for outcome in outcomes],

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/examples/demo.ipynb

@@ -2,6 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
+   "id": "7fb27b941602401d91542211134fc71a",
    "metadata": {},
    "source": [
     "# clef-compactor demo\n",
@@ -12,10 +13,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "acae54e37e7d407bbb7b55eff062a284",
    "metadata": {},
    "outputs": [],
    "source": [
     "import httpx\n",
+    "\n",
     "from clef_compactor import ClefCompactor\n",
     "from clef_compactor.client import ClefClient\n",
     "from clef_compactor.config import Settings\n",
@@ -50,6 +53,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -71,6 +75,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "8dd0d8092fe74a7c96281538738b07e2",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -83,6 +88,7 @@
   },
   {
    "cell_type": "markdown",
+   "id": "72eea5119410473aa328ad9291626812",
    "metadata": {},
    "source": [
     "## The compacted context\n",
@@ -93,6 +99,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "8edb47106e1a46a883d545849b8ab81b",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -103,6 +110,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "10185d26023b46108eb7d9f57d49d2b3",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -116,6 +124,7 @@
   },
   {
    "cell_type": "markdown",
+   "id": "8763a12b2bbd4a93a75aff182afb95dc",
    "metadata": {},
    "source": [
     "## Going live\n",
@@ -130,8 +139,15 @@
   }
  ],
  "metadata": {
-  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
-  "language_info": {"name": "python", "version": "3.10"}
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.10"
+  }
  },
  "nbformat": 4,
  "nbformat_minor": 5

clef_compactor-0.2.1/kaggle-kernel/kernel-metadata.json

@@ -0,0 +1,13 @@
+{
+  "id": "gjusev/clef-compactor-evals",
+  "title": "clef-compactor-evals",
+  "code_file": "script.py",
+  "language": "python",
+  "kernel_type": "script",
+  "is_private": true,
+  "enable_gpu": true,
+  "enable_internet": true,
+  "dataset_sources": [],
+  "kernel_sources": [],
+  "model_sources": []
+}

clef_compactor-0.2.1/kaggle-kernel/script.py

@@ -0,0 +1,47 @@
+"""Kaggle kernel: measure clef-compactor against the open-weights Clef model.
+
+Follows the verified laya-evals kernel pattern (github.com/Gjusev/laya-evals):
+clone the public repo, pip install it, run the measurement, print the verdict.
+Outputs land in /kaggle/working/ and come back with `kaggle kernels output`.
+
+Hardware: Kaggle T4 x2 (2 x 16 GB). We run Cloudflare/clef-flash (9B) in
+float16 sharded across both GPUs with device_map="auto"; the 27B model does
+not fit on this hardware. Credentials are never needed and never embedded:
+the local backend loads weights from Hugging Face.
+"""
+
+import subprocess
+import sys
+
+
+def run(cmd: str) -> None:
+    print(f"$ {cmd}", flush=True)
+    subprocess.run(cmd, shell=True, check=True)
+
+
+# 0. GPU sanity: Kaggle can silently boot without the GPU you asked for.
+print("=== GPU check ===", flush=True)
+run("nvidia-smi")
+
+# 1. Repo + dependencies. transformers 5.10.2 is the version the Clef release
+#    was tested with; Qwen3.5 support needs the 5.x line.
+run("git clone --depth 1 https://github.com/Gjusev/clef-compactor.git")
+run(f"{sys.executable} -m pip install -q --no-input ./clef-compactor")
+run(f"{sys.executable} -m pip install -q --no-input 'transformers==5.10.2' accelerate safetensors")
+
+run(f"{sys.executable} -c 'import torch, transformers; "
+    "print(torch.__version__, torch.cuda.is_available(), transformers.__version__)'")
+
+# 2. The measurement: full pipeline (batching, ranking, budget) against the
+#    locally loaded clef-flash weights, scored on the committed gold dataset.
+run(
+    f"{sys.executable} clef-compactor/evals/run_eval.py "
+    "--mode local --model-path Cloudflare/clef-flash --dtype float16 "
+    "--out /kaggle/working/results-local.json --compare-laya"
+)
+
+# 3. Optional stretch: the 27B model in float16 needs ~54 GB and does not fit
+#    on 2 x T4; skipped on purpose. If Kaggle ever offers bigger GPUs, run:
+#    run_eval.py --mode local --model-path Cloudflare/clef
+
+print("KERNEL COMPLETE", flush=True)

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "clef-compactor"
-version = "0.2.0"
+version = "0.2.1"
 description = "Query-aware RAG context compaction using Cloudflare's Clef. Keep the evidence, cut the noise."
 readme = "README.md"
 license = { text = "Apache-2.0" }

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/src/clef_compactor/__init__.py

@@ -29,7 +29,7 @@ from .exceptions import (
 )
 from .models import ClefReply, CompactResult, DropReason, ScoredChunk, TokenUsage
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 __all__ = [
     "AsyncClefCompactor",

{clef_compactor-0.2.0 → clef_compactor-0.2.1}/tests/test_smoke.py

@@ -8,7 +8,9 @@ from conftest import envelope, noul_answer
 
 
 def test_package_exposes_version() -> None:
-    assert clef_compactor.__version__ == "0.2.0"
+    from importlib.metadata import version
+
+    assert clef_compactor.__version__ == version("clef-compactor")
 
 
 def test_public_api_surface() -> None: