fusion-runtime 0.1.2__tar.gz → 0.1.4__tar.gz

{fusion_runtime-0.1.2/fusion_runtime.egg-info → fusion_runtime-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fusion-runtime
-Version: 0.1.2
+Version: 0.1.4
 Summary: Self-hosted voice agent runtime: speech-to-text, LLM and text-to-speech on your own GPU, with tools
 Author-email: Samarth Urs <hello@fusion-runtime.dev>
 License-Expression: Apache-2.0
@@ -81,6 +81,9 @@ Requires Python 3.11–3.13.
 pip install fusion-runtime
 ```
 
+The first install can sit quiet for a few minutes: where there's no ready-made llama.cpp for your
+machine, pip builds it. That's normal.
+
 An agent is one file. This is the whole thing:
 
 ```python
@@ -177,6 +180,32 @@ calling on), llama-server (`--jinja`) or a hosted API. The in-process
 llama.cpp runtime can't, and `frun up` says so at startup. A runnable version is
 [`examples/tools_agent.py`](examples/tools_agent.py).
 
+### Answering from your own documents
+
+Retrieval is a tool like any other: a function that takes the caller's question and returns a few
+short passages, which the model answers from.
+
+```python
+@tool(timeout_s=3)
+def search_help(question: str) -> dict:
+    """Search the shop's help articles: returns, refunds, delivery, cancelling orders.
+
+    Args:
+        question: What the caller wants to know, in their words.
+    """
+    return {"passages": my_search(question, limit=3)}
+```
+
+`my_search` is yours: a vector database, Elasticsearch, your help centre's API.
+[`examples/knowledge_agent.py`](examples/knowledge_agent.py) is a runnable version with a small
+keyword search over the Markdown files in [`examples/knowledge/`](examples/knowledge/), in plain
+Python with nothing to install.
+
+On a call, two things matter more than in a chat window. A lookup costs a second model round
+before the caller hears anything, so if your knowledge fits in a page or two, put it in the prompt
+instead: that's faster than any search. And passages go into the prompt, so keep them short (a few
+sentences each, a few per search): long ones slow the model's first word.
+
 ## On your own site
 
 ```html
@@ -191,6 +220,42 @@ site embeds. With no `url` it connects back to wherever the script came from.
 Browsers only allow a microphone on `https://`, so a deployment needs TLS and `wss://`. A page
 never holds an API key: your backend mints it a short-lived token.
 
+## Phone calls
+
+Keep your own number and your provider account (and its KYC); the provider streams each call to
+fusion, and the agent answers it. Twilio, Exotel and Plivo:
+
+| Provider | Point the number at | Set where fusion runs |
+|---|---|---|
+| Twilio | voice webhook "A call comes in": `https://<server>/v1/phone/twilio` (POST) | `FUSION_TWILIO_AUTH_TOKEN`: the account's auth token |
+| Exotel | a Voicebot applet in the call flow: `wss://fusion:<key>@<server>/v1/phone/exotel/stream?sample-rate=16000` | `FUSION_ACCEPTED_KEYS`, with the key in that URL |
+| Plivo | the number's answer URL: `https://<server>/v1/phone/plivo` (POST) | `FUSION_PLIVO_AUTH_TOKEN`: the account's auth token |
+
+Each request is checked before a call is taken: Twilio's webhook signature, the key in Exotel's
+URL, Plivo's signature on the stream. Behind a proxy, set `FUSION_PUBLIC_URL` to the address the
+provider calls (on a Runpod pod `frun up` sets it), since that's the address they sign.
+
+A call gets everything a browser conversation does: the same limits, interruption (the provider
+drops the agent's unplayed audio the moment the caller talks over it), tools and logs, with
+`transport=twilio` and the call's id on `session.start`. On top of that:
+
+- The greeting is made when the server starts, so a caller hears it within ~50 ms.
+- A full server answers "all our lines are busy" rather than dropping the call.
+- A dead line is faint hiss, not silence: nothing from the caller above `FUSION_PHONE_FLOOR_RMS`
+  (40) for `FUSION_DEAD_LINE_S` (10 s) is logged as a quiet line, and every call's `session.end`
+  records its loudest moment and whether the caller was heard at all.
+
+Try it without a number or an account: [`tools/phone_sim`](tools/phone_sim/README.md) plays the
+provider's side from your machine, with your microphone, at phone quality. All three are built
+from the providers' documentation and tested with that simulator, not yet on real numbers; if your
+account behaves differently, open an issue.
+
+On Twilio, test with an upgraded account and a number of your own. A trial account's calls come
+from a shared sandbox number that neither signs its requests nor streams the call's audio to a
+server, and a newly upgraded account may hold calls until Twilio's KYC (a Trust Hub customer
+profile) is approved. A refused webhook is logged with the address fusion checked, which tells a
+wrong address from a wrong token.
+
 ## Authentication
 
 ```bash
@@ -241,7 +306,7 @@ console), so you can reproduce them rather than trusting ours. Barge-in fired on
 ## Several callers at once
 
 Measured on the same 3090, real WebSocket sessions, three turns each
-([`scripts/concurrency_check.py`](scripts/concurrency_check.py)). Response is the server's own
+(`frun bench`). Response is the server's own
 figure: the turn ends, audio comes back.
 
 | Callers | In-process llama.cpp | vLLM | SGLang |
@@ -287,9 +352,13 @@ fusion where it is:
 ```bash
 python3 -m venv ~/vllm-env && ~/vllm-env/bin/pip install vllm
 export FUSION_LLM_ENGINE_ENV=~/vllm-env
+frun models pull agent.py     # the LLM too (5.6 GB here), where vLLM looks for it
 frun up agent.py
 ```
 
+Without the pull, vLLM downloads the model during its first start, which takes 5–15 minutes and
+looks like nothing is happening.
+
 `frun up` starts the server before loading speech, so it takes its share of the GPU first (60%
 by default, which fits a 4-bit 7B model next to Whisper and Kokoro on a 24 GB card), restarts it
 if it exits, and stops it with Ctrl+C. GPU memory, context length, tool parsing, extra engine

{fusion_runtime-0.1.2 → fusion_runtime-0.1.4}/README.md

@@ -27,6 +27,9 @@ Requires Python 3.11–3.13.
 pip install fusion-runtime
 ```
 
+The first install can sit quiet for a few minutes: where there's no ready-made llama.cpp for your
+machine, pip builds it. That's normal.
+
 An agent is one file. This is the whole thing:
 
 ```python
@@ -123,6 +126,32 @@ calling on), llama-server (`--jinja`) or a hosted API. The in-process
 llama.cpp runtime can't, and `frun up` says so at startup. A runnable version is
 [`examples/tools_agent.py`](examples/tools_agent.py).
 
+### Answering from your own documents
+
+Retrieval is a tool like any other: a function that takes the caller's question and returns a few
+short passages, which the model answers from.
+
+```python
+@tool(timeout_s=3)
+def search_help(question: str) -> dict:
+    """Search the shop's help articles: returns, refunds, delivery, cancelling orders.
+
+    Args:
+        question: What the caller wants to know, in their words.
+    """
+    return {"passages": my_search(question, limit=3)}
+```
+
+`my_search` is yours: a vector database, Elasticsearch, your help centre's API.
+[`examples/knowledge_agent.py`](examples/knowledge_agent.py) is a runnable version with a small
+keyword search over the Markdown files in [`examples/knowledge/`](examples/knowledge/), in plain
+Python with nothing to install.
+
+On a call, two things matter more than in a chat window. A lookup costs a second model round
+before the caller hears anything, so if your knowledge fits in a page or two, put it in the prompt
+instead: that's faster than any search. And passages go into the prompt, so keep them short (a few
+sentences each, a few per search): long ones slow the model's first word.
+
 ## On your own site
 
 ```html
@@ -137,6 +166,42 @@ site embeds. With no `url` it connects back to wherever the script came from.
 Browsers only allow a microphone on `https://`, so a deployment needs TLS and `wss://`. A page
 never holds an API key: your backend mints it a short-lived token.
 
+## Phone calls
+
+Keep your own number and your provider account (and its KYC); the provider streams each call to
+fusion, and the agent answers it. Twilio, Exotel and Plivo:
+
+| Provider | Point the number at | Set where fusion runs |
+|---|---|---|
+| Twilio | voice webhook "A call comes in": `https://<server>/v1/phone/twilio` (POST) | `FUSION_TWILIO_AUTH_TOKEN`: the account's auth token |
+| Exotel | a Voicebot applet in the call flow: `wss://fusion:<key>@<server>/v1/phone/exotel/stream?sample-rate=16000` | `FUSION_ACCEPTED_KEYS`, with the key in that URL |
+| Plivo | the number's answer URL: `https://<server>/v1/phone/plivo` (POST) | `FUSION_PLIVO_AUTH_TOKEN`: the account's auth token |
+
+Each request is checked before a call is taken: Twilio's webhook signature, the key in Exotel's
+URL, Plivo's signature on the stream. Behind a proxy, set `FUSION_PUBLIC_URL` to the address the
+provider calls (on a Runpod pod `frun up` sets it), since that's the address they sign.
+
+A call gets everything a browser conversation does: the same limits, interruption (the provider
+drops the agent's unplayed audio the moment the caller talks over it), tools and logs, with
+`transport=twilio` and the call's id on `session.start`. On top of that:
+
+- The greeting is made when the server starts, so a caller hears it within ~50 ms.
+- A full server answers "all our lines are busy" rather than dropping the call.
+- A dead line is faint hiss, not silence: nothing from the caller above `FUSION_PHONE_FLOOR_RMS`
+  (40) for `FUSION_DEAD_LINE_S` (10 s) is logged as a quiet line, and every call's `session.end`
+  records its loudest moment and whether the caller was heard at all.
+
+Try it without a number or an account: [`tools/phone_sim`](tools/phone_sim/README.md) plays the
+provider's side from your machine, with your microphone, at phone quality. All three are built
+from the providers' documentation and tested with that simulator, not yet on real numbers; if your
+account behaves differently, open an issue.
+
+On Twilio, test with an upgraded account and a number of your own. A trial account's calls come
+from a shared sandbox number that neither signs its requests nor streams the call's audio to a
+server, and a newly upgraded account may hold calls until Twilio's KYC (a Trust Hub customer
+profile) is approved. A refused webhook is logged with the address fusion checked, which tells a
+wrong address from a wrong token.
+
 ## Authentication
 
 ```bash
@@ -187,7 +252,7 @@ console), so you can reproduce them rather than trusting ours. Barge-in fired on
 ## Several callers at once
 
 Measured on the same 3090, real WebSocket sessions, three turns each
-([`scripts/concurrency_check.py`](scripts/concurrency_check.py)). Response is the server's own
+(`frun bench`). Response is the server's own
 figure: the turn ends, audio comes back.
 
 | Callers | In-process llama.cpp | vLLM | SGLang |
@@ -233,9 +298,13 @@ fusion where it is:
 ```bash
 python3 -m venv ~/vllm-env && ~/vllm-env/bin/pip install vllm
 export FUSION_LLM_ENGINE_ENV=~/vllm-env
+frun models pull agent.py     # the LLM too (5.6 GB here), where vLLM looks for it
 frun up agent.py
 ```
 
+Without the pull, vLLM downloads the model during its first start, which takes 5–15 minutes and
+looks like nothing is happening.
+
 `frun up` starts the server before loading speech, so it takes its share of the GPU first (60%
 by default, which fits a 4-bit 7B model next to Whisper and Kokoro on a 24 GB card), restarts it
 if it exits, and stops it with Ctrl+C. GPU memory, context length, tool parsing, extra engine

{fusion_runtime-0.1.2 → fusion_runtime-0.1.4}/fusion_runtime/agent.py

@@ -220,7 +220,8 @@ class Agent:
     vad: Optional[VAD] = None
     tools: Sequence[Any] = ()  # functions the model can call; plain functions become tools
     greeting: Optional[str] = None  # said when a caller connects, before they speak; None = wait for them
-    profile: str = "development"  # the defaults everything above is applied to
+    greeting_interruptible: bool = True  # False: always heard in full ("this call may be recorded")
+    profile: str = "development"  # the defaults everything above is applied to: development (laptops), production (GPU)
     source: Optional[Path] = None  # the file it was loaded from, when it came from one
 
     def __post_init__(self) -> None:
@@ -248,10 +249,10 @@ class Agent:
 
     def config(self, environ: Optional[Mapping[str, str]] = None):
         """This agent as a PipelineConfig: profile defaults, the agent on top, then the environment."""
-        from fusion_runtime.config import PROFILES, with_env_overrides
+        from fusion_runtime.config import PROFILES, profile_choices, with_env_overrides
 
         if self.profile not in PROFILES:
-            raise AgentError(f"unknown profile {self.profile!r}; choose one of: {', '.join(PROFILES)}")
+            raise AgentError(f"unknown profile {self.profile!r}; choose one of: {profile_choices()}")
         config = PROFILES[self.profile]
         parts: Dict[str, Any] = {}
 

{fusion_runtime-0.1.2 → fusion_runtime-0.1.4}/fusion_runtime/catalog/download.py

@@ -45,7 +45,7 @@ def bytes_to_download(entry: ModelEntry, root: Path, force: bool = False) -> int
     return sum(entry.files[n] for n in names)
 
 
-def check_disk_space(needed: int, root: Path) -> None:
+def check_disk_space(needed: int, root: Path, setting: str = "FUSION_MODEL_DIR") -> None:
     probe = root
     while not probe.exists():  # model dir may not exist yet
         probe = probe.parent
@@ -54,7 +54,7 @@ def check_disk_space(needed: int, root: Path) -> None:
         raise NotEnoughDiskSpace(
             f"Need {format_size(needed)} (plus {format_size(DISK_HEADROOM_BYTES)} headroom) "
             f"but only {format_size(free)} is free at {probe}. "
-            "Free up space or set FUSION_MODEL_DIR to a bigger disk."
+            f"Free up space or set {setting} to a bigger disk."
         )
 
 
@@ -189,16 +189,83 @@ def pull_hf(ref: str, root: Path, log: Callable[[str], None] = print, force: boo
     except DownloadError:
         raise
     except Exception as e:
-        name = type(e).__name__
-        if name in ("GatedRepoError", "RepositoryNotFoundError") or "401" in str(e) or "403" in str(e):
-            raise ModelAccessDenied(
-                f"{repo} is gated or private. Accept its terms on huggingface.co/{repo}, then put an access "
-                f"token in ${HF_TOKEN_ENV} (export {HF_TOKEN_ENV}=hf_...)"
-            ) from e
-        raise DownloadError(f"{repo}: download failed: {name}: {e}") from e
+        raise _hub_error(repo, e) from e
     return target
 
 
+def _hub_error(repo: str, e: Exception) -> DownloadError:
+    name = type(e).__name__
+    if name in ("GatedRepoError", "RepositoryNotFoundError") or "401" in str(e) or "403" in str(e):
+        return ModelAccessDenied(
+            f"{repo} is gated or private. Accept its terms on huggingface.co/{repo}, then put an access "
+            f"token in ${HF_TOKEN_ENV} (export {HF_TOKEN_ENV}=hf_...)")
+    return DownloadError(f"{repo}: download failed: {name}: {e}")
+
+
+# ---- models vLLM and SGLang load -------------------------------------------------------------
+#
+# A model on vLLM or SGLang is loaded by the engine, from Hugging Face's own cache (HF_HOME), not
+# from fusion's model directory. Left to the engine, a first `frun up` spends 5-15 minutes
+# downloading inside its start-up, looking hung. `frun models pull agent.py` fetches it into that
+# same cache beforehand, with progress, so the engine finds it there.
+
+# Weights in other formats the engines don't load (a repo may ship several), and training leftovers
+ENGINE_UNWANTED = HF_UNWANTED + ["*.gguf", "*.onnx", "*.onnx_data", "onnx/**", "*.pt"]
+
+
+def engine_files(repo: str, revision: Optional[str] = None, token: Optional[str] = None) -> list:
+    """[(name, size)] of what vLLM or SGLang needs from a repo: weights, configs, tokenizer, chat template."""
+    from huggingface_hub import HfApi
+
+    info = HfApi().repo_info(repo, revision=revision, files_metadata=True, token=token)
+    files = [(f.rfilename, f.size or 0) for f in (info.siblings or [])
+             if not any(Path(f.rfilename).match(p) for p in ENGINE_UNWANTED)]
+    if any(name.endswith(".safetensors") for name, _ in files):
+        files = [f for f in files if not f[0].endswith(".bin")]  # the same weights again, in the older format
+    return files
+
+
+def in_engine_cache(repo: str, filename: str = "config.json", revision: Optional[str] = None) -> bool:
+    """Whether Hugging Face's cache holds this file of the repo. Looks on disk only, never online."""
+    from huggingface_hub import try_to_load_from_cache
+
+    return isinstance(try_to_load_from_cache(repo, filename, revision=revision), str)
+
+
+def engine_cache_dir() -> Path:
+    from huggingface_hub import constants
+
+    return Path(constants.HF_HUB_CACHE)
+
+
+def pull_for_engine(ref: str, log: Callable[[str], None] = print, force: bool = False) -> int:
+    """Download a vLLM/SGLang model into Hugging Face's cache. Returns the bytes fetched (0: all there)."""
+    repo, revision, filename = hf_reference(ref if ref.startswith(HF_PREFIX) else HF_PREFIX + ref)
+    if filename:
+        raise DownloadError(f"vLLM and SGLang load a whole repo; name it as hf:{repo}, without {filename!r}")
+    from huggingface_hub import snapshot_download
+
+    token = os.getenv(HF_TOKEN_ENV)
+    try:
+        files = engine_files(repo, revision, token)
+        if not files:
+            raise DownloadError(f"{repo} has no files vLLM or SGLang can load")
+        missing = files if force else [f for f in files if not in_engine_cache(repo, f[0], revision)]
+        if not missing:
+            return 0
+        needed = sum(size for _, size in missing)
+        cache = engine_cache_dir()
+        check_disk_space(needed, cache, setting="HF_HOME")
+        log(f"  {repo}{'@' + revision if revision else ''} ({format_size(needed)}) → {cache}, where the engine looks")
+        snapshot_download(repo_id=repo, revision=revision, allow_patterns=[name for name, _ in files],
+                          token=token, force_download=force)
+    except DownloadError:
+        raise
+    except Exception as e:
+        raise _hub_error(repo, e) from e
+    return needed
+
+
 def _files_to_fetch(repo: str, revision: Optional[str], filename: Optional[str], token) -> list:
     """The named file plus the small files every copy needs, or the whole repo when it holds one model."""
     files = hf_files(repo, revision, token)

{fusion_runtime-0.1.2 → fusion_runtime-0.1.4}/fusion_runtime/cli/_checks.py

@@ -235,7 +235,7 @@ def check_acceleration() -> List[CheckResult]:
     if gpu_offload:
         results.append(CheckResult(OK, f"llama.cpp can use the GPU ({backend})"))
         if DEVELOPMENT_CONFIG.llm.n_gpu_layers == 0:
-            results.append(CheckResult(INFO, "The development profile still runs the LLM on CPU (n_gpu_layers=0)"))
+            results.append(CheckResult(INFO, "The development profile is for laptops: it runs the LLM on the CPU (n_gpu_layers=0); production uses the GPU"))
     elif cuda_devices:
         # The GPU image builds llama.cpp for the CPU on purpose: on an NVIDIA card the LLM
         # belongs on a model server, which serves about three times the callers.
@@ -259,20 +259,27 @@ def check_models() -> List[CheckResult]:
     from fusion_runtime.config import model_dir, model_dir_source
 
     results = [CheckResult(INFO, f"Model directory: {short_path(model_dir())} ({model_dir_source()})")]
+    # Each profile's missing models count only on the machine that profile is for: on a GPU server,
+    # the laptop profile's CPU models aren't a problem, and the reverse.
+    gpu = bool(_cuda_device_count())
     dev_missing = missing_models(Profile.development, apply_env=False)
     if dev_missing:
-        results.append(CheckResult(FAIL, f"development profile is missing {', '.join(dev_missing)}",
-                                   "frun models pull"))
+        results.append(CheckResult(
+            INFO if gpu else FAIL,
+            f"development profile (laptops) is missing {', '.join(dev_missing)}"
+            + (" (only for --config development)" if gpu else ""),
+            "frun models pull",
+        ))
     else:
-        results.append(CheckResult(OK, "development profile: all models installed"))
+        results.append(CheckResult(OK, "development profile (laptops): all models installed"))
 
     prod_missing = missing_models(Profile.production, apply_env=False)
     if not prod_missing:
-        results.append(CheckResult(OK, "production profile: all models installed"))
+        results.append(CheckResult(OK, "production profile (NVIDIA GPU): all models installed"))
     else:
         results.append(CheckResult(
-            WARN if _cuda_device_count() else INFO,
-            f"production profile is missing {', '.join(prod_missing)}",
+            WARN if gpu else INFO,
+            f"production profile (NVIDIA GPU) is missing {', '.join(prod_missing)}",
             "frun models pull --config production",
         ))
     return results

{fusion_runtime-0.1.2 → fusion_runtime-0.1.4}/fusion_runtime/cli/app.py

@@ -1,6 +1,7 @@
 """The Typer app every `frun` command registers on."""
 import typer
 
+from fusion_runtime.cli.bench import bench
 from fusion_runtime.cli.doctor import doctor
 from fusion_runtime.cli.keys import key_app, keys_app, token
 from fusion_runtime.cli.models import models_app
@@ -49,6 +50,7 @@ def _root(
 
 app.command()(up)
 app.command()(talk)
+app.command()(bench)
 app.add_typer(key_app, name="key")
 app.add_typer(keys_app, name="keys")
 app.command()(token)