fast-mode-kit 0.2.0__py3-none-any.whl

fast_mode_kit-0.2.0.dist-info/METADATA

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: fast-mode-kit
+Version: 0.2.0
+Summary: Starter kit for adding a fast, data-only answer mode to any chat agent
+License: MIT
+Keywords: llm,agents,latency,tool-calling,evals
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.6
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Dynamic: license-file
+
+# fast-mode-kit
+
+A starter kit for adding a **Fast mode** to any chat agent. Fast mode answers exactly what was asked, from your data, right the first time, in a few seconds.
+
+It comes from shipping Fast mode in a live product. Every rule here traces back to a measured result or a real failure, and the [failure catalogue](docs/08-failure-catalogue.md) lists them.
+
+It works with **frontier APIs** (any OpenAI-compatible endpoint) and with **models you host yourself** (mlx, vLLM, Ollama, llama.cpp).
+
+## Quick start (offline, no key, no cost)
+
+```bash
+pip install fast-mode-kit
+
+fastkit demo "rank products by revenue"     # the example shop agent, offline
+fastkit eval                                # accuracy, latency, cost and fallback report
+fastkit init my-agent                       # your own Fast mode project
+cd my-agent && python agent.py "how many widgets in stock?"
+```
+
+The offline stand-in model picks tools by word overlap and has no understanding, so the example scores about 40% on `fastkit eval`. That is expected: it shows what keyword routing gets you. A real model fills the gap.
+
+## With a real model
+
+```bash
+export FASTKIT_API_KEY=...          # leave unset for local servers (Ollama, vLLM, mlx, llama.cpp)
+
+# 1. One tiny request: is this model fast enough, and does it call tools?
+fastkit check --base-url https://openrouter.ai/api/v1 --model google/gemini-2.5-flash
+
+# 2. Measure your agent on your questions, with a spend cap
+fastkit eval --agent agent.py --gold gold.jsonl \
+    --base-url https://openrouter.ai/api/v1 --model google/gemini-2.5-flash \
+    --price-in 0.30 --price-out 2.50 --spend-cap 0.50
+```
+
+`fastkit check` prints PASS, SLOW or FAIL against the 4.5 s first-words budget and tells you whether the model called the tool. A model that answers without calling it can't run the one-call loop.
+
+`fastkit.checks.safe_filter` runs after the answer has finished. Use it in evals and logs, or buffer the output if you want to filter before the user sees it.
+
+## Develop
+
+```bash
+git clone <this repo> && cd fast-mode-kit
+pip install -e ".[dev]" && pytest -q        # every test runs against a mock model
+```
+
+## What's inside
+
+| Path | What it is |
+|---|---|
+| `docs/` | the guide, chapters 1-9: what Fast is, architecture, deadlines, prompts, checks, model choice, self-hosting, failure catalogue, measuring |
+| `fastkit/templates/` | prompt templates: Fast system, reader, writer, server-owned lines |
+| `fastkit/client.py` | streaming client for any OpenAI-compatible server |
+| `fastkit/loop.py` | the one-call loop: read, call tools and write in one conversation, with a fallback |
+| `fastkit/pipeline.py` | the fallback: a reader plans, the server fetches, a writer answers |
+| `fastkit/hedge.py` | hedged routes; a stream that has started is never cut |
+| `fastkit/tools.py` | tool registry, `find_tools`, `note_turn`, propose/confirm writes |
+| `fastkit/checks.py` | do-no-harm figure checks |
+| `fastkit/modes.py` | Fast-only gate |
+| `fastkit/mock.py` | scripted model for offline tests |
+| `fastkit/examples/shop/` | a complete agent over a toy shop database |
+| `fastkit/evals/` | gold-set runner, report and spend cap |
+| `fastkit/cli.py` | the `fastkit` command: demo, init, check, eval |
+| `fastkit/scaffold/` | the starter project `fastkit init` writes |
+
+## Adapt it to your domain in 5 steps
+
+1. **Tools:** write read tools that return rows with every derived figure already computed (totals, ranks, percentages), plus propose tools for writes. See `fastkit/examples/shop/tools.py`.
+2. **Prompt:** render `fastkit/templates/fast_system.md` with your domain and tool list.
+3. **Loop:** call `fast_answer(...)` with your model and a pipeline fallback. See `fastkit/examples/shop/agent.py`.
+4. **Gold set:** write 40-60 real questions with their expected figures, across the languages your users write in.
+5. **Measure:** run `fastkit eval` on two or three models and pick by tail latency and per-language accuracy, not by the median.
+
+## The rules in one screen
+
+- No second chance in Fast, so the first answer must be right.
+- One line per asked part, figure first, data only.
+- The server computes every figure. The model copies.
+- Fall back to another model path, never to keyword guesses.
+- A deadline never cuts an answer that has started streaming.
+- A check that edits a correct line is a bug in the check.
+- Keep Fast changes behind one gate, and prove the other modes unchanged.
+- Never judge a reading model with thinking switched off.
+
+## License
+
+MIT

fast_mode_kit-0.2.0.dist-info/RECORD

@@ -0,0 +1,39 @@
+fast_mode_kit-0.2.0.dist-info/licenses/LICENSE,sha256=h6zptfibmvmQwYvAXPsLKTwaUHN6kacQq_fZ9xR-zPw,1083
+fastkit/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+fastkit/__main__.py,sha256=E6Gls0DNz8GQK2K-kOUIx8cYhgANW_CH54VKrfCfs14,52
+fastkit/checks.py,sha256=-u6_6EikWLq9XmudtyRmZaz_vmetMg9FSjWbYK8sRyM,4687
+fastkit/cli.py,sha256=brKOrCYLCkqfqPPqeF_uOG-_TOMA91Qp8ItNE32a__c,7760
+fastkit/client.py,sha256=RpVdi01ENdGb2WrQlFMcjxf5T-9ktLL6Ra16ycZmPXM,3604
+fastkit/hedge.py,sha256=0M9JVlV_4POBRNlvVTm87PuHhmaX-uNY_atTy2r-6H8,3139
+fastkit/loop.py,sha256=9a8yKegvKYQj8WMnGoOeFhWw8I94zaLY_LGVPR8Qtyo,5966
+fastkit/mock.py,sha256=s-2HEyb2YUN5Ty8ap-4xEs_ovxBHm8PxHRJVgJPgKzE,1250
+fastkit/modes.py,sha256=c0CDePFgnapWbGuw5cFZAprGBvh5ZcJknNoDU9dZKwQ,651
+fastkit/pipeline.py,sha256=c2ioZItlx__VW_VUbT-44A3EpeFn-MIawIOYSfVO61A,2021
+fastkit/prompts.py,sha256=ZBRpA9Y0ElrjA5uHwPjlerIpjkoBwy8KvMByLYpWHIw,916
+fastkit/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+fastkit/tools.py,sha256=9zO5uKmJiD5Nnus9cPO9nXDERANIAqV7dpQm1tw5AzY,2453
+fastkit/types.py,sha256=iLOY_IvRHyI_5i7NFSGp5mXIZeWvZsa2jxmkjfuYiJI,1371
+fastkit/evals/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+fastkit/evals/report.py,sha256=lleMizxkwfRoFsvVV8NMDoZxB-BnvgKW9hO8mC8jVgU,1904
+fastkit/evals/run.py,sha256=he4IlBJ5BuQCnuyXmMXg2BJ8xO_CdRvXqhW7IjWu7Ho,3863
+fastkit/evals/schema.py,sha256=CUn_qXfDms_qoeoPACKmJ3IGRMZ_sBzQJwepNlJv2Tg,1300
+fastkit/evals/shop_gold.jsonl,sha256=soI_JRtB6mHyQgaxGvx8am4044IkjEi8TQDIwV2Q1hI,3476
+fastkit/examples/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+fastkit/examples/shop/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+fastkit/examples/shop/agent.py,sha256=r6MeyDCiIMepXe96-m_0McBAVBHz92W-ULiKKRu5PWM,4133
+fastkit/examples/shop/data.py,sha256=yoDLFDiz9-DA86fp0V5BrKgfVglh3HscfZLPtHBwRuw,1642
+fastkit/examples/shop/demo_model.py,sha256=CV-T6zFC2aQoYOu5n8XuxtyIuo7IvSrMu1AGMsAHciI,3069
+fastkit/examples/shop/tools.py,sha256=rbdznsCmOqr1JsZFr0H9qIFROHe7B2StlgrCenTvAgo,4287
+fastkit/scaffold/README.md,sha256=nXtS7JwRHfCKifnSPT263fHJku59x1ZeJk8k6Alsiq0,606
+fastkit/scaffold/agent.py,sha256=Pc1mHo-t2PCAg5B54XCjor7nCn9brmKATO5qENmeaNE,3627
+fastkit/scaffold/gold.jsonl,sha256=R1C-UXPGSLEIv077-AJxG6u8LGOcBwQUusjsy5VESzk,378
+fastkit/scaffold/tools.py,sha256=Un6euayO7NACdiLahRSYK5-3ckyj7JMg7dFz5vsCp-w,1475
+fastkit/templates/fast_system.md,sha256=m9cl84Vk_k9vb9-TSfgplWcWAtukXzzWVP8oOtJGER8,1390
+fastkit/templates/notes.md,sha256=oxLTDpYGuxMktuyuv-xn2HLvHkj-uOwAmEmOnje1g1E,366
+fastkit/templates/reader.md,sha256=tFYlSIqcMoWLErQ9Jh1FgAHMPLa-f56rgWvV_EyV5Lc,414
+fastkit/templates/writer.md,sha256=9pe5XJVD08TsTx62tyNDAAxs7jUcggATN0Ry6K6ia68,309
+fast_mode_kit-0.2.0.dist-info/METADATA,sha256=H3l1hCE2ejFAB5_TTrRECqHKz88LWx59O5XZDq2BgKM,5168
+fast_mode_kit-0.2.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+fast_mode_kit-0.2.0.dist-info/entry_points.txt,sha256=S9R4Ha1DdsZ-FdqTLXELBR4uRboaB-GBxDnj5Jxcenc,45
+fast_mode_kit-0.2.0.dist-info/top_level.txt,sha256=ublpjaGX1NVzyRCpSCqknBi8ZOI0cEcOBGDZmVKRf2s,8
+fast_mode_kit-0.2.0.dist-info/RECORD,,

fast_mode_kit-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

fast_mode_kit-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+fastkit = fastkit.cli:main

fast_mode_kit-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 fast-mode-kit contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fast_mode_kit-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+fastkit

fastkit/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())

fastkit/checks.py

@@ -0,0 +1,131 @@
+"""Do-no-harm figure checks.
+
+Rules learned the hard way:
+- an identifier that appears in the rows is sourced, even if it has digits;
+- a zero or negative that answers the question is said, never trimmed;
+- a sentence is dropped only when it has an unsourced figure AND no sourced
+  one; a check never leaves an empty answer or a fragment.
+Every edit a check makes to a correct line is a bug in the check.
+"""
+from __future__ import annotations
+
+import logging
+import re
+from typing import Any, Iterable
+
+log = logging.getLogger(__name__)
+
+_NUM = re.compile(r"(?<![\w.])-?\d[\d,]*(?:\.\d+)?%?")
+_RANGE_DASH = re.compile(r"(?<=\d)-(?=\d)")
+_LIST_MARK = re.compile(r"^\s*\d+[.)]\s+")
+_TOKEN = re.compile(r"[A-Za-z0-9][A-Za-z0-9_\-./]*[A-Za-z0-9]|[A-Za-z0-9]")
+_SENT = re.compile(r"(?<=[.!?])\s+")
+
+
+def _walk(obj: Any) -> Iterable[Any]:
+    if isinstance(obj, dict):
+        for v in obj.values():
+            yield from _walk(v)
+    elif isinstance(obj, (list, tuple)):
+        for v in obj:
+            yield from _walk(v)
+    else:
+        yield obj
+
+
+def _row_facts(rows: list[dict], question: str) -> tuple[list[float], set[str]]:
+    nums: list[float] = []
+    strings: set[str] = set()
+    for v in _walk(rows):
+        if isinstance(v, bool):
+            continue
+        if isinstance(v, (int, float)):
+            nums.append(float(v))
+        elif isinstance(v, str):
+            strings.add(v.lower())
+            try:
+                nums.append(float(v.replace(",", "")))
+            except ValueError:
+                pass
+    for m in _NUM.finditer(question):
+        nums.append(float(m.group(0).rstrip("%").replace(",", "")))
+    strings.add(question.lower())
+    return nums, strings
+
+
+def _matches(text: str, nums: list[float]) -> bool:
+    pct = text.endswith("%")
+    raw = text.rstrip("%").replace(",", "")
+    try:
+        val = float(raw)
+    except ValueError:
+        return False
+    decimals = len(raw.split(".")[1]) if "." in raw else 0
+    for n in nums:
+        for cand in ((n, n * 100) if pct else (n,)):
+            for c in (cand, abs(cand)) if val >= 0 else (cand,):
+                # "fell 3.2%" states -3.2 without its sign
+                if round(c, decimals) == val or c == val:
+                    return True
+                if decimals == 0 and abs(c) >= 1 and round(c) == val:
+                    return True
+    return False
+
+
+def _figures(answer: str) -> list[tuple[str, bool]]:
+    """(figure, is_identifier) pairs found in the answer."""
+    out = []
+    spans = []
+    for tok in _TOKEN.finditer(answer):
+        t = tok.group(0)
+        if not any(c.isdigit() for c in t):
+            continue
+        # codes (SKU-1042), dates (2026-11-01), versions: identifiers, not quantities
+        if re.fullmatch(r"\d+(?:\.\d+)?-\d+(?:\.\d+)?", t):
+            continue  # a range like 10-20: two plain numbers, handled below
+        if any(c.isalpha() for c in t) or "-" in t.lstrip("-") or t.count(".") > 1 or "/" in t:
+            out.append((t, True))
+            spans.append(tok.span())
+    for m in _NUM.finditer(_RANGE_DASH.sub(" ", answer)):
+        if any(a <= m.start() < b for a, b in spans):
+            continue
+        out.append((m.group(0).rstrip(".,"), False))
+    return out
+
+
+def ungrounded_figures(answer: str, rows: list[dict], question: str = "") -> list[str]:
+    nums, strings = _row_facts(rows, question)
+    bad = []
+    for fig, ident in _figures(answer):
+        if ident:
+            if not any(fig.lower() in s for s in strings):
+                bad.append(fig)
+        elif not _matches(fig, nums):
+            bad.append(fig)
+    return bad
+
+
+def safe_filter(answer: str, rows: list[dict], question: str = "") -> str:
+    """Drop a sentence only when every figure in it is unsourced. Lines and
+    list markers are kept as written; never returns an empty answer."""
+    out_lines = []
+    dropped = False
+    for line in answer.split("\n"):
+        mark = _LIST_MARK.match(line)
+        prefix, body = (line[:mark.end()], line[mark.end():]) if mark else ("", line)
+        kept = []
+        for sent in _SENT.split(body):
+            figs = [f for f, _ in _figures(sent)]
+            bad = ungrounded_figures(sent, rows, question)
+            if bad and len(bad) == len(figs):
+                log.info("dropped unsourced sentence: %r", sent)
+                dropped = True
+                continue
+            kept.append(sent)
+        if kept or not body.strip():
+            out_lines.append(prefix + " ".join(kept))
+    result = "\n".join(out_lines).strip()
+    if not result:
+        log.warning("check would empty the answer; keeping original")
+        return answer
+    return result if dropped else answer

fastkit/cli.py

@@ -0,0 +1,190 @@
+"""fastkit command line.
+
+    fastkit demo "top 3 products by revenue"     answer one question with the example agent
+    fastkit init my-agent                         start a Fast mode project of your own
+    fastkit check --base-url URL --model NAME     one tiny request: speed and tool calling
+    fastkit eval [--agent agent.py] [--gold g]    accuracy, latency, cost, fallback report
+
+Nothing here calls a paid API unless you pass --base-url.
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import importlib.util
+import os
+import shutil
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from .types import Message, Model
+
+HERE = Path(__file__).resolve().parent
+KEY_ENV = "FASTKIT_API_KEY"
+
+PROBE_TOOL = {"type": "function", "function": {
+    "name": "get_time", "description": "Current time. Call this for any question about the time.",
+    "parameters": {"type": "object", "properties": {}}}}
+
+
+def _client(base_url: str, model: str, key_env: str) -> Model:
+    from .client import ChatClient
+    return ChatClient(base_url, os.environ.get(key_env), model)
+
+
+async def probe(model: Model, timeout: float = 30.0) -> dict[str, Any]:
+    """Send one tiny tool-call request. Time the first event and see if a tool was called."""
+    msgs = [Message(role="system", content="Use tools for facts. Never answer from memory."),
+            Message(role="user", content="What time is it?")]
+    t0 = time.monotonic()
+    first = None
+    called = False
+    error = None
+    text: list[str] = []
+    try:
+        async def go():
+            nonlocal first, called
+            async for ev in model.stream(msgs, tools=[PROBE_TOOL]):
+                if ev.kind in ("token", "tool_call") and first is None:
+                    first = time.monotonic() - t0
+                if ev.kind == "tool_call":
+                    called = True
+                elif ev.kind == "token":
+                    text.append(ev.text)
+                elif ev.kind == "error":
+                    raise RuntimeError(ev.text)
+        await asyncio.wait_for(go(), timeout)
+    except Exception as e:  # report, never crash the check
+        error = f"{type(e).__name__}: {e}"
+    return {"first_event_s": first, "total_s": time.monotonic() - t0, "called_tool": called,
+            "text": "".join(text), "error": error}
+
+
+def verdict(rep: dict[str, Any]) -> str:
+    if rep["error"]:
+        return f"FAIL: request failed ({rep['error']})"
+    lines = []
+    if not rep["called_tool"]:
+        lines.append("FAIL: answered without calling the tool. A one-call Fast loop needs a model that calls tools; "
+                     "use the pipeline path or another model.")
+    f = rep["first_event_s"]
+    if f is None:
+        lines.append("FAIL: no output.")
+    elif f > 4.5:
+        lines.append(f"SLOW: first event after {f:.2f} s, over the 4.5 s first-words budget. Try another provider route or a smaller model.")
+    elif f > 2.5:
+        lines.append(f"OK but tight: first event {f:.2f} s. Little room for a second round.")
+    else:
+        lines.append(f"GOOD: first event {f:.2f} s.")
+    if rep["called_tool"] and not lines[0].startswith(("FAIL", "SLOW")):
+        lines.insert(0, "PASS: called the tool.")
+    return "\n".join(lines)
+
+
+def cmd_check(a) -> int:
+    rep = asyncio.run(probe(_client(a.base_url, a.model, a.api_key_env)))
+    print(verdict(rep))
+    print(f"[first event {rep['first_event_s'] or 0:.2f}s, total {rep['total_s']:.2f}s]")
+    return 0 if rep["called_tool"] and not rep["error"] else 1
+
+
+def cmd_demo(a) -> int:
+    from .examples.shop.agent import make_registry, run
+    reg = make_registry()
+    if a.base_url:
+        model = _client(a.base_url, a.model, a.api_key_env)
+    else:
+        from .examples.shop.demo_model import DemoModel, demo_vocab
+        model = DemoModel(reg, vocab=demo_vocab())
+        print("(offline stand-in model: it only matches words to tools. Pass --base-url for a real model.)\n")
+    out = asyncio.run(run(a.question, a.mode, model, registry=reg))
+    print(out["text"] or out["confirm"] or "(no answer)")
+    print(f"\n[first token {out['first_token_s'] or 0:.2f}s, total {out['total_s']:.2f}s, "
+          f"fallback={out['fell_back']}, tools={out['tools_called']}]")
+    return 0
+
+
+def cmd_init(a) -> int:
+    dest = Path(a.path)
+    if dest.exists() and any(dest.iterdir()):
+        print(f"{dest} is not empty; pick a new folder.", file=sys.stderr)
+        return 1
+    dest.mkdir(parents=True, exist_ok=True)
+    for f in (HERE / "scaffold").iterdir():
+        if f.is_file() and f.suffix in (".py", ".md", ".jsonl"):
+            shutil.copy(f, dest / f.name)
+    shutil.copytree(HERE / "templates", dest / "prompts")
+    print(f"Created {dest}. Next:\n  cd {dest}\n  python agent.py \"how many widgets in stock?\"\n"
+          f"Then read {dest / 'README.md'}.")
+    return 0
+
+
+def _load_agent(path: str):
+    p = Path(path).resolve()
+    sys.path.insert(0, str(p.parent))
+    spec = importlib.util.spec_from_file_location("user_agent", p)
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+def cmd_eval(a) -> int:
+    from .evals.report import markdown
+    from .evals.run import client_agent, load_gold, mock_agent, run_eval
+    gold = a.gold or (None if a.agent else str(HERE / "evals" / "shop_gold.jsonl"))
+    if not gold:
+        print("--gold is required with --agent", file=sys.stderr)
+        return 2
+    if a.agent:
+        mod = _load_agent(a.agent)
+        model = mod.make_model(a.base_url, a.model, a.api_key_env)
+        agent = lambda q: mod.answer(q, model)  # noqa: E731
+    elif a.base_url:
+        agent = client_agent(a.base_url, a.model, a.api_key_env, a.mode)
+    else:
+        agent = mock_agent()
+    price = (a.price_in, a.price_out) if a.price_in is not None and a.price_out is not None else None
+    if a.base_url and price is None:
+        print("note: no --price-in/--price-out, so cost shows $0 and the spend cap cannot stop the run")
+    results = asyncio.run(run_eval(load_gold(gold), agent, spend_cap_usd=a.spend_cap, price_per_mtok=price))
+    print(markdown(results))
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="fastkit", description="Build and measure a Fast answer mode.")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    def model_args(p, need_url=False):
+        p.add_argument("--base-url", required=need_url, help="OpenAI-compatible endpoint, e.g. http://localhost:11434/v1")
+        p.add_argument("--model", default="")
+        p.add_argument("--api-key-env", default=KEY_ENV, help=f"env var holding the key (default {KEY_ENV})")
+
+    p = sub.add_parser("demo", help="answer one question with the example shop agent")
+    p.add_argument("question")
+    p.add_argument("--mode", default="fast")
+    model_args(p)
+    p.set_defaults(fn=cmd_demo)
+
+    p = sub.add_parser("init", help="create a starter Fast mode project")
+    p.add_argument("path")
+    p.set_defaults(fn=cmd_init)
+
+    p = sub.add_parser("check", help="one tiny request: first-event time and tool calling")
+    model_args(p, need_url=True)
+    p.set_defaults(fn=cmd_check)
+
+    p = sub.add_parser("eval", help="run a gold set and print the report")
+    p.add_argument("--agent", help="your agent.py (from fastkit init); default: the example shop")
+    p.add_argument("--gold")
+    p.add_argument("--mode", default="fast")
+    p.add_argument("--price-in", type=float, help="USD per million input tokens")
+    p.add_argument("--price-out", type=float, help="USD per million output tokens")
+    p.add_argument("--spend-cap", type=float, default=0.50)
+    model_args(p)
+    p.set_defaults(fn=cmd_eval)
+
+    a = ap.parse_args(argv)
+    return a.fn(a)

fastkit/client.py

@@ -0,0 +1,77 @@
+"""Streaming client for any OpenAI-compatible chat endpoint.
+
+Works with hosted APIs and local servers (mlx_lm.server, vLLM, Ollama,
+llama.cpp). Use ``extra_body`` for server-specific fields, for example
+``{"adapters": "/abs/path"}`` so mlx_lm.server applies your LoRA adapter.
+"""
+from __future__ import annotations
+
+import json
+from typing import Any, AsyncIterator
+
+import httpx
+
+from .types import Message, StreamEvent, ToolCall
+
+
+class ChatClient:
+    def __init__(self, base_url: str, api_key: str | None, model: str,
+                 extra_body: dict[str, Any] | None = None, timeout: float = 60.0,
+                 transport: httpx.AsyncBaseTransport | None = None):
+        self.base_url = base_url.rstrip("/")
+        self.api_key = api_key
+        self.model = model
+        self.extra_body = extra_body or {}
+        self.timeout = timeout
+        self._transport = transport
+
+    async def stream(self, messages: list[Message], tools: list[dict] | None = None,
+                     **kw: Any) -> AsyncIterator[StreamEvent]:
+        body: dict[str, Any] = {
+            "model": self.model,
+            "messages": [m.to_api() for m in messages],
+            "stream": True,
+            "stream_options": {"include_usage": True},
+            **self.extra_body, **kw,
+        }
+        if tools:
+            body["tools"] = tools
+        headers = {"Authorization": f"Bearer {self.api_key}"} if self.api_key else {}
+        partial: dict[int, dict[str, Any]] = {}
+        usage = None
+        async with httpx.AsyncClient(timeout=self.timeout, transport=self._transport) as http:
+            async with http.stream("POST", f"{self.base_url}/chat/completions",
+                                   json=body, headers=headers) as resp:
+                if resp.status_code >= 400:
+                    text = (await resp.aread()).decode(errors="replace")[:300]
+                    yield StreamEvent(kind="error", text=f"HTTP {resp.status_code}: {text}")
+                    return
+                async for line in resp.aiter_lines():
+                    if not line.startswith("data:"):
+                        continue
+                    data = line[5:].strip()
+                    if data == "[DONE]":
+                        break
+                    chunk = json.loads(data)
+                    usage = chunk.get("usage") or usage
+                    for choice in chunk.get("choices") or []:
+                        delta = choice.get("delta") or {}
+                        if delta.get("content"):
+                            yield StreamEvent(kind="token", text=delta["content"])
+                        for tc in delta.get("tool_calls") or []:
+                            slot = partial.setdefault(tc.get("index", 0),
+                                                      {"id": "", "name": "", "args": ""})
+                            slot["id"] = tc.get("id") or slot["id"]
+                            fn = tc.get("function") or {}
+                            slot["name"] = fn.get("name") or slot["name"]
+                            slot["args"] += fn.get("arguments") or ""
+        for i in sorted(partial):
+            s = partial[i]
+            try:
+                args = json.loads(s["args"]) if s["args"] else {}
+            except json.JSONDecodeError:
+                yield StreamEvent(kind="error", text=f"bad tool arguments for {s['name']}")
+                continue
+            yield StreamEvent(kind="tool_call",
+                              tool_call=ToolCall(id=s["id"] or f"call_{i}", name=s["name"], arguments=args))
+        yield StreamEvent(kind="done", usage=usage)

fastkit/evals/report.py

@@ -0,0 +1,48 @@
+"""Markdown report: accuracy, latency spread, cost, fallback rate."""
+from __future__ import annotations
+
+from collections import defaultdict
+
+from .schema import Result
+
+
+def _pct(values: list[float], p: float) -> float:
+    s = sorted(values)
+    if not s:
+        return 0.0
+    k = (len(s) - 1) * p
+    lo, hi = int(k), min(int(k) + 1, len(s) - 1)
+    return s[lo] + (s[hi] - s[lo]) * (k - lo)
+
+
+def markdown(results: list[Result]) -> str:
+    n = len(results) or 1
+    ft = [r.first_token_s for r in results if r.first_token_s is not None]
+    tot = [r.total_s for r in results]
+    rows = [
+        ("questions", str(len(results))),
+        ("accuracy", f"{round(100 * sum(r.correct for r in results) / n)}%"),
+        ("median first token", f"{_pct(ft, .5):.2f} s"),
+        ("p95 first token", f"{_pct(ft, .95):.2f} s"),
+        ("worst first token", f"{max(ft, default=0):.2f} s"),
+        ("median total", f"{_pct(tot, .5):.2f} s"),
+        ("total cost", f"${sum(r.cost_usd for r in results):.4f}"),
+        ("fallback rate", f"{round(100 * sum(1 for r in results if r.fell_back) / n)}%"),
+        ("answers with unsourced figures", str(sum(1 for r in results if r.ungrounded))),
+    ]
+    by_lang: dict[str, list[Result]] = defaultdict(list)
+    for r in results:
+        by_lang[r.language].append(r)
+    for lang, rs in sorted(by_lang.items()):
+        rows.append((f"accuracy ({lang})", f"{sum(r.correct for r in rs)}/{len(rs)}"))
+    reasons: dict[str, int] = defaultdict(int)
+    for r in results:
+        if r.fell_back:
+            reasons[r.fell_back] += 1
+    for k, v in sorted(reasons.items()):
+        rows.append((f"fallback: {k}", str(v)))
+    lines = ["| measure | value |", "|---|---|", *[f"| {a} | {b} |" for a, b in rows]]
+    misses = [r.id for r in results if not r.correct]
+    if misses:
+        lines += ["", "Missed: " + ", ".join(misses)]
+    return "\n".join(lines)