cre-router 0.1.0__tar.gz

cre_router-0.1.0/.gitignore

@@ -0,0 +1,36 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+venv/
+
+# Secrets
+.env
+.env.*
+
+# Tooling
+.pytest_cache/
+.ruff_cache/
+.ipynb_checkpoints/
+
+# Experiment outputs
+artifacts/
+logs/
+results/
+*.log
+
+# Downloaded datasets (fetch with data/download.py; not redistributed here)
+data/*.jsonl
+
+# Model training outputs (e.g. cre qe-train --output-dir qe-aime)
+qe-*/
+*.safetensors
+*.pt
+
+# OS / editors
+.DS_Store
+.idea/
+.vscode/

cre_router-0.1.0/DEVELOP.md

@@ -0,0 +1,26 @@
+# Development
+
+Local setup for working on the package and running its tests.
+
+```bash
+git clone https://github.com/ymoslem/CRE-Router
+cd CRE-Router
+pip install -e ".[qe,serve,dev]"
+pytest
+```
+
+The `dev` extra adds the test dependencies (`pytest`, plus `fastapi` and
+`httpx` for the serve smoke test); `qe` and `serve` bring the runtime the
+tests exercise.
+
+The default `pytest` run is CPU-only and needs no model weights. Two smoke
+tests that require real resources are marked `integration` and skipped unless
+you point them at a GPU/server via environment variables, e.g.:
+
+```bash
+# QE classifier load + predict
+CRE_TEST_QE_CHECKPOINT=ymoslem/ModernBERT-base-AIME-1983-2023-instruct-qe-classifier-binary-10ep-lr5e-05 \
+  pytest -m integration
+# vLLM measurement (needs a running `vllm serve <model>`)
+CRE_TEST_VLLM_MODEL=WeiboAI/VibeThinker-1.5B pytest -m integration
+```

cre_router-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

cre_router-0.1.0/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.4
+Name: cre-router
+Version: 0.1.0
+Summary: Cluster, Route, Escalate: a cascaded framework for cost-aware LLM serving
+Project-URL: Homepage, https://github.com/ymoslem/CRE-Router
+Project-URL: Repository, https://github.com/ymoslem/CRE-Router
+Project-URL: Issues, https://github.com/ymoslem/CRE-Router/issues
+Author: Yasmin Moslem
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: cascades,litellm,llm,model-serving,quality-estimation,routing,vllm
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: anyio>=4.0
+Requires-Dist: numpy>=1.26
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: scikit-learn>=1.4
+Requires-Dist: sentence-transformers>=3.0
+Provides-Extra: dev
+Requires-Dist: fastapi>=0.115; extra == 'dev'
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: eval
+Requires-Dist: vllm>=0.19; extra == 'eval'
+Provides-Extra: full
+Requires-Dist: accelerate>=1.0; extra == 'full'
+Requires-Dist: datasets>=3.0; extra == 'full'
+Requires-Dist: fastapi>=0.115; extra == 'full'
+Requires-Dist: httpx>=0.27; extra == 'full'
+Requires-Dist: litellm>=1.60; extra == 'full'
+Requires-Dist: torch>=2.4; extra == 'full'
+Requires-Dist: transformers>=4.57; extra == 'full'
+Requires-Dist: uvicorn>=0.30; extra == 'full'
+Requires-Dist: vllm>=0.19; extra == 'full'
+Provides-Extra: qe
+Requires-Dist: accelerate>=1.0; extra == 'qe'
+Requires-Dist: datasets>=3.0; extra == 'qe'
+Requires-Dist: torch>=2.4; extra == 'qe'
+Requires-Dist: transformers>=4.57; extra == 'qe'
+Provides-Extra: serve
+Requires-Dist: fastapi>=0.115; extra == 'serve'
+Requires-Dist: httpx>=0.27; extra == 'serve'
+Requires-Dist: litellm>=1.60; extra == 'serve'
+Requires-Dist: torch>=2.4; extra == 'serve'
+Requires-Dist: transformers>=4.57; extra == 'serve'
+Requires-Dist: uvicorn>=0.30; extra == 'serve'
+Description-Content-Type: text/markdown
+
+# CRE-Router
+
+Implementation of the paper [Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving](https://arxiv.org/abs/2606.27457).
+
+Production LLM serving trades accuracy against cost. CRE-Router routes each
+query to the most cost-effective model in a pool, then escalates low-quality
+outputs to a stronger model:
+
+- **Stage 1 (clustering-based routing).** Queries are embedded and clustered
+  offline; each cluster is routed to the model that minimizes a cost-adjusted
+  error score (Error + lambda * Cost), with lambda tuned once to satisfy a
+  latency (TPOT) budget.
+- **Stage 2 (quality-estimation cascade).** A lightweight ModernBERT
+  classifier inspects each efficient-model output and escalates low-quality
+  answers up an ordered ladder of stronger models.
+
+Both stages train only on task-correctness labels obtainable from standard
+benchmark evaluation; no extra annotation is required.
+
+## Install
+
+```bash
+pip install "cre-router[full]"
+```
+
+`full` installs the whole pipeline on one machine. Narrower installs are
+available through the `serve`, `qe`, and `eval` extras; see the
+[installation guide](https://github.com/ymoslem/CRE-Router#installation).
+
+## Usage
+
+The workflow is driven by the `cre` CLI, one stage per step:
+
+`cre cluster` → `cre evaluate` → `cre fit` → `cre qe-train` → `cre serve`
+
+Run `cre <stage> --help` for options. Runnable end-to-end quickstarts (a
+no-GPU routing-table demo and a full serving walkthrough) are in the
+[README](https://github.com/ymoslem/CRE-Router#readme).
+
+## Links
+
+- **Source and documentation:** https://github.com/ymoslem/CRE-Router
+- **Paper:** https://arxiv.org/abs/2606.27457
+- **Reproducing the paper:** https://github.com/ymoslem/CRE-Router/blob/main/REPRODUCE.md
+
+Apache-2.0 licensed.

cre_router-0.1.0/PYPI.md

@@ -0,0 +1,46 @@
+# CRE-Router
+
+Implementation of the paper [Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving](https://arxiv.org/abs/2606.27457).
+
+Production LLM serving trades accuracy against cost. CRE-Router routes each
+query to the most cost-effective model in a pool, then escalates low-quality
+outputs to a stronger model:
+
+- **Stage 1 (clustering-based routing).** Queries are embedded and clustered
+  offline; each cluster is routed to the model that minimizes a cost-adjusted
+  error score (Error + lambda * Cost), with lambda tuned once to satisfy a
+  latency (TPOT) budget.
+- **Stage 2 (quality-estimation cascade).** A lightweight ModernBERT
+  classifier inspects each efficient-model output and escalates low-quality
+  answers up an ordered ladder of stronger models.
+
+Both stages train only on task-correctness labels obtainable from standard
+benchmark evaluation; no extra annotation is required.
+
+## Install
+
+```bash
+pip install "cre-router[full]"
+```
+
+`full` installs the whole pipeline on one machine. Narrower installs are
+available through the `serve`, `qe`, and `eval` extras; see the
+[installation guide](https://github.com/ymoslem/CRE-Router#installation).
+
+## Usage
+
+The workflow is driven by the `cre` CLI, one stage per step:
+
+`cre cluster` → `cre evaluate` → `cre fit` → `cre qe-train` → `cre serve`
+
+Run `cre <stage> --help` for options. Runnable end-to-end quickstarts (a
+no-GPU routing-table demo and a full serving walkthrough) are in the
+[README](https://github.com/ymoslem/CRE-Router#readme).
+
+## Links
+
+- **Source and documentation:** https://github.com/ymoslem/CRE-Router
+- **Paper:** https://arxiv.org/abs/2606.27457
+- **Reproducing the paper:** https://github.com/ymoslem/CRE-Router/blob/main/REPRODUCE.md
+
+Apache-2.0 licensed.

cre_router-0.1.0/README.md

@@ -0,0 +1,251 @@
+# CRE-Router
+
+Implementation of the paper, [**Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving**](https://arxiv.org/abs/2606.27457).
+
+
+## System overview
+
+Production deployment of large language models forces a trade-off between
+accuracy and cost. This framework routes each query to the most
+cost-effective model in a pool, then escalates low-quality outputs to a
+stronger model:
+
+- **Stage 1 (clustering-based routing).** Queries are embedded and
+  clustered offline; each cluster is assigned to the model minimizing
+  `Error(m, c) + lambda * Cost_norm(m)`, where `lambda` ($\lambda$, the cost weight) is
+  tuned once to `lambda*` ($\lambda^*$) to satisfy a Time Per Output Token (TPOT) budget.
+- **Stage 2 (quality-estimation cascade).** A lightweight classifier
+  inspects each efficient-model output and escalates low-quality answers to
+  a stronger model. With more than two models, escalation follows an ordered
+  ladder (weakest to strongest, following the $\lambda$ table): each model has
+  its own classifier and the router rolls up to the next stronger model until
+  an output is accepted or the top is reached.
+
+Both stages train only on task-correctness labels obtainable from standard
+benchmark evaluation; no extra annotation is required.
+
+<p align="center"><img src="img/system.svg" alt="Two-stage cascaded routing system" width="640"></p>
+
+## Pipeline
+
+The full workflow is driven by the `cre` CLI, one stage per step:
+
+`cre cluster` → `cre evaluate` → `cre fit` → `cre qe-train` → `cre serve`
+
+Serving (`cre serve`) switches between the live vLLM backends through [LiteLLM](https://github.com/BerriAI/litellm).
+
+| Stage | What it does | Input | Output | Key options |
+|---|---|---|---|---|
+| `cre cluster` | Embeds training queries and fits k-means centroids (k chosen by Silhouette) | JSONL of training queries | `centroids.npy` and `router.json`; `train_assignments.jsonl` | `--k`, `--embedding-model` |
+| `cre evaluate` | Runs each model per cluster through vLLM's benchmark, scores answers, averages per-cluster error and TPOT. This is the slowest stage; cost scales with model size, output length, and `--runs`, and it runs once per model. | dataset JSONL, a running vLLM server, fitted centroids | per-model entry in the stats JSON; raw runs under `results/` | `--runs`, `--port`, `--concurrency` |
+| `cre fit` | Pareto-prunes the pool, sweeps $\lambda$, selects $\lambda^*$ under the TPOT budget | stats JSON, budget B | routing table and $\lambda^*$ in `router.json` | `--budget` (required), `--output` |
+| `cre qe-train` | Fine-tunes ModernBERT-base as the accept/escalate QE classifier | HF dataset of model outputs with correctness labels | QE classifier checkpoint | `--learning-rate`, `--max-length`, `--attn-implementation` |
+| `cre serve` | Runs the live router: sends each incoming query to its cluster's assigned model, and escalates weak answers to a stronger model | serving config, running backends | live HTTP router on port 4000 | `--config`, `--port` |
+
+Full flags for any stage: `cre <stage> --help`.
+
+## Installation
+
+```bash
+pip install "cre-router[full]"
+```
+
+This is the whole pipeline on one machine: clustering and routing, the
+cascade router, QE classifier training/evaluation, and vLLM for measurement
+and for hosting backend models. If you want a narrower install, pick from the
+extras below.
+
+| Extra | Adds | For |
+|---|---|---|
+| (core) | numpy, scikit-learn, sentence-transformers, pyyaml, anyio | `cre cluster`, `cre fit` — always installed |
+| `serve` | + litellm, fastapi, uvicorn, httpx, torch, transformers | `cre serve` — the Stage 1 + Stage 2 cascade router (loads ModernBERT in-process) |
+| `qe` | + torch, transformers, datasets, accelerate | `cre qe-train`, `cre qe-eval` — train and evaluate the QE classifier |
+| `eval` | + vllm | `cre evaluate` — per-cluster measurement; also provides `vllm serve` for backends |
+| `full` | qe + serve + eval | the whole pipeline on one machine |
+
+The vLLM backends the router talks to are separate processes started with
+`vllm serve <model>` (cf.
+[Quickstart: serve CRE-Router](#quickstart-serve-cre-router-gpu-required)); a
+`serve`-only host still needs vLLM installed wherever those backends run.
+
+Efficient QE training also needs [FlashAttention](https://github.com/Dao-AILab/flash-attention)
+(`flash-attn`), installed as a second step once torch is already present
+(e.g. after `pip install "cre-router[qe]"`):
+
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+(the paper used `flash-attn==2.8.3`). The flag matters: without it, pip's
+isolated build environment can't see your installed torch/CUDA, so the build
+targets the wrong version. Without flash-attn at all, pass
+`cre qe-train --attn-implementation sdpa` to fall back.
+
+## Quickstart: routing table (no GPU)
+
+A standalone, zero-setup demo of the routing math itself, no serving and no
+GPU involved. Running it reproduces the paper's routing table and $\lambda^*$
+selection directly from the checked-in stats, so you can see how Stage 1
+decides which model handles which cluster before setting up any backends.
+It is *not* a prerequisite for the end-to-end GPU-based serving,
+which fits its own routing table as one of its steps (cf.
+[Quickstart: serve CRE-Router](#quickstart-serve-cre-router-gpu-required)).
+The per-cluster
+stats measured in the paper are checked in under
+[`configs/`](configs), so the routing table and budgeted $\lambda^*$ reproduce
+without any GPU:
+
+```bash
+cre fit --stats configs/aime_stats.json --budget 20
+```
+
+This prints the Pareto analysis, the $\lambda$ sweep (routing regions), and the
+$\lambda^*$ selection. To regenerate the stats from scratch (clustering plus
+per-model measurement on a GPU), run `cre cluster` and `cre evaluate` on your
+own dataset and models, the same commands used for the paper (see the
+[Pipeline](#pipeline) table above). If you want to reproduce the paper's exact
+numbers with its datasets and models, see [REPRODUCE.md](REPRODUCE.md) instead.
+
+## Quickstart: serve CRE-Router (GPU required)
+
+This walks through standing up the live router end to end, using the paper's
+AIME pool as a concrete, runnable example. Swap in your own models, dataset,
+and config the same way once you see the shape of it.
+
+1. Start one vLLM server per pool model. `--max-model-len` must cover input +
+   the task's generation length (40,960 for AIME), or long outputs truncate:
+
+   ```bash
+   vllm serve WeiboAI/VibeThinker-1.5B --port 8001 --max-model-len 42000
+   vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 --port 8002 --max-model-len 42000
+   ```
+
+2. Fit centroids and write the routing table into an artifacts directory.
+   Reusing the checked-in stats skips the GPU measurement step:
+
+   ```bash
+   python data/download.py --dataset ymoslem/AIME-clustered --split train --output data/aime_train.jsonl
+   cre cluster --input data/aime_train.jsonl --embeddings-field embeddings --output artifacts/aime
+   cre fit --stats configs/aime_stats.json --budget 20 --output artifacts/aime
+   ```
+
+3. Write a serving config. It carries only deployment wiring — the model pool
+   (each model's endpoint) and the QE classifier checkpoints. The routing
+   table, $\lambda^*$, and the escalation ladder are read from the artifacts and
+   the arithmetic, never set by hand. Point
+   [`example_config_aime24.yaml`](src/cre_router/server/example_config_aime24.yaml)
+   at your backends and serve (a TeleQnA config,
+   [`example_config_teleqna.yaml`](src/cre_router/server/example_config_teleqna.yaml),
+   is also provided):
+
+   ```bash
+   cre serve --config example_config_aime24.yaml
+   ```
+
+4. Send a standard chat-completions request:
+
+   ```bash
+   curl http://localhost:4000/v1/chat/completions \
+     -H "Content-Type: application/json" \
+     -d '{"messages": [{"role": "user", "content": "..."}]}'
+   ```
+
+   Routing decisions are returned as `x-cre-cluster`, `x-cre-stage1-model`,
+   `x-cre-final-model`, `x-cre-escalated`, and `x-cre-path` response headers.
+   Streaming is not supported: Stage 2 must inspect the complete response before
+   deciding whether to escalate it.
+
+5. Monitor the deployment. `GET /stats` returns live tallies (total requests,
+   escalation rate, and counts per cluster, per final model, and per
+   escalation path). Setting `decision_log: <path>` in the config also appends
+   one JSON record per request to that file. Both are designed to add no
+   latency to requests: `/stats` is in-memory counters, and the decision log
+   is written by a background task fed through a non-blocking queue.
+
+## Supported models and datasets
+
+**Models.** Any model you can serve behind a chat-completions HTTP endpoint
+(the paper uses vLLM). The pool is arbitrary and can mix sizes and families;
+routing, Pareto pruning, and the escalation ladder adapt to whatever pool you
+measure. The paper's pools are Qwen 3 / Qwen 3.5, Gemma 4, and VibeThinker
+(see [REPRODUCE.md](REPRODUCE.md)).
+
+**Datasets.** `cre evaluate` ships two tasks, `aime` (numeric answers) and
+`teleqna` (multiple choice), each defining an answer parser and sampling
+preset. A new dataset with a different answer format needs one new `Task`
+entry (a parser + sampling) in [evaluate.py](src/cre_router/evaluate.py);
+clustering, routing, and the QE cascade are domain-agnostic and need no
+changes.
+
+## Data formats
+
+Inputs are JSONL, one object per line. Cluster ids are strings ("0", "1", ...)
+throughout.
+
+- **Training queries** (`cre cluster`): `prompt` (text to embed). Any other
+  fields are ignored.
+- **Evaluation dataset** (`cre evaluate`): `prompt` (sent to the model),
+  `answer` (ground truth), and optionally `cluster`. Without `cluster`, pass
+  `--artifacts` and each row is assigned to its nearest fitted centroid.
+- **QE dataset** (`cre qe-train`, `cre qe-eval`): `question`, `full_output`
+  (the model's completion), `num_tokens` (output length), and `decision_label`
+  (1 = accept, 0/2 = escalate). Matches the released `ymoslem/*-router` datasets.
+- **Model stats** (`cre fit`): JSON with `cluster_sizes` and per-model `errors`
+  and `cluster_tpot_ms`; see [`configs/aime_stats.json`](configs/aime_stats.json).
+- **Serving config** (`cre serve`): YAML; see
+  [`example_config_aime24.yaml`](src/cre_router/server/example_config_aime24.yaml).
+
+An artifacts directory (written by `cre cluster` / `cre fit`) holds
+`centroids.npy`, `router.json` (embedding model, routing table, $\lambda^*$), and
+`train_assignments.jsonl`.
+
+## Repository layout
+
+```
+cre-router/
+├── src/cre_router/
+│   ├── clustering.py            Stage 1: embed queries, fit k-means centroids
+│   ├── routing.py               Stage 1: cost-aware routing, Pareto, lambda*
+│   ├── evaluate.py              Stage 1: measure per-cluster accuracy and TPOT via vLLM
+│   ├── artifacts.py             load/save centroids + routing table
+│   ├── cli.py                   the `cre` entry point
+│   ├── qe/                      Stage 2: QE classifier
+│   │   ├── train.py             fine-tune the accept/escalate classifier
+│   │   ├── classifier.py        inference wrapper used by the router
+│   │   └── evaluate.py          standalone QE metrics (`cre qe-eval`)
+│   └── server/                  live cascade router
+│       ├── cascade_router.py    Stage 1 routing + Stage 2 escalation ladder
+│       ├── app.py               HTTP endpoint, /stats, decision log
+│       ├── example_config_aime24.yaml    serving config (AIME)
+│       └── example_config_teleqna.yaml   serving config (TeleQnA)
+├── configs/                     checked-in per-model stats for `cre fit`
+├── data/download.py             fetch released datasets from the Hugging Face Hub
+├── img/system.svg               architecture figure
+├── tests/                       unit tests (routing math vs paper, cascade, ...)
+├── requirements-paper.txt       frozen environment behind the paper's numbers
+├── REPRODUCE.md                 reproducing the paper
+├── DEVELOP.md                   local development and running the tests
+└── README.md
+```
+
+## Citation
+
+```bibtex
+@article{moslem2026clusterrouteescalate,
+      title={Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving}, 
+      author={Yasmin Moslem and Magdalena Kacmajor and Vasudevan Nedumpozhimana and Ammar Abbas and Solmaz Panahi and David Lynch and Zhuangzhuang Nie and Alexandros Agapitos and Aleksandar Milenovic and Hongmeng Song and Yucheng Shi and Yue Pan and Patricia Buffini and John D. Kelleher},
+      year={2026},
+      eprint={2606.27457},
+      archivePrefix={arXiv},
+      primaryClass={cs.PF},
+      url={https://arxiv.org/abs/2606.27457}, 
+}
+```
+
+## Development
+
+For local setup and running the test suite, see [DEVELOP.md](DEVELOP.md).
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE).