cortexgrid-infer 0.3.7__tar.gz → 0.3.9__tar.gz

{cortexgrid_infer-0.3.7 → cortexgrid_infer-0.3.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid-infer
-Version: 0.3.7
+Version: 0.3.9
 Summary: Serve apps per task (text-to-text, text-to-image, image-to-mesh) and importers per source (HuggingFace) for running models on cortexgrid
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid-infer
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid-infer
@@ -9,7 +9,7 @@ License-File: LICENSE
 Requires-Python: <3.12,>=3.11
 Requires-Dist: accelerate>=0.33
 Requires-Dist: anthropic>=0.40.0
-Requires-Dist: cortexgrid>=0.3.12
+Requires-Dist: cortexgrid>=0.3.18
 Requires-Dist: diffusers>=0.38
 Requires-Dist: fastapi>=0.133
 Requires-Dist: google-genai>=1.0
@@ -23,6 +23,7 @@ Requires-Dist: sentencepiece>=0.2
 Requires-Dist: torch>=2.0.0
 Requires-Dist: transformers>=4.47.0
 Provides-Extra: dev
+Requires-Dist: parameterized>=0.9.0; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Description-Content-Type: text/markdown
 
@@ -50,23 +51,24 @@ Upgrading from 0.2.x? See [MIGRATION.md](https://github.com/robodatalab/cortexgr
 A model is two things, and the library keeps them apart:
 
 - **What runs it** — a serve app in `cortexgrid_infer.serve_apps`, named for the task
-  it runs, not for where the model came from. It loads the weights, answers the task's
-  routes (defined in `cortexgrid_infer.protocols`), and names the client that speaks
-  them. One serve app runs every model of its task, whichever source staged it.
+  it runs, not for where the model came from. It loads the weights and serves the
+  task's endpoints, declared by the task's type (`CompletingModel`, say); cortexgrid
+  generates the client that calls them. One serve app runs every model of its task,
+  whichever source staged it.
 - **Where its weights come from** — an importer in `cortexgrid_infer.importers`, one
   per source. It names the model in the registry, downloads the weights, and sizes
   them from the source's metadata. It is handed the serve app, so one importer takes
   any model its source holds.
 
-| Serve app | Task | Runs | Client |
+| Serve app | Task | Runs | Task type |
 |---|---|---|---|
-| `Text2Text` | text to text | any causal LM in the transformers layout, via `AutoModelForCausalLM` | `ServedCompletingModel` (`complete`) |
-| `Text2Image` | text (and image) to image | any diffusers pipeline | `ServedGeneratingModel` (`generate`) |
-| `Image2Mesh` — a base: subclass it with the model's `load` and `make_mesh` | image to mesh | the model the subclass loads, behind the task's `POST /mesh` | `ServedMeshingModel` (`mesh`: one picture in, a `GeneratedMesh` out) |
-| `TextRewriter` | text to text, rewritten | any encoder-decoder LM (T5, BART, Marian, ...) in the transformers layout, via `AutoModelForSeq2SeqLM` | `ServedRewritingModel` (`rewrite`: one text in, one text out) |
-| `AnthropicText2Text` | text to text | the Anthropic API, forwarded from the cluster | `ServedCompletingModel` (`complete`) |
-| `GeminiText2Text` | text to text | the Gemini API, forwarded from the cluster | `ServedCompletingModel` (`complete`) |
-| `GeminiText2Image` | text to image | the Gemini API, forwarded from the cluster | `ServedGeneratingModel` (`generate`) |
+| `Text2Text` | text to text | any causal LM in the transformers layout, via `AutoModelForCausalLM` | `CompletingModel` (`complete`), plus its own `loglikelihoods` and `last_hidden_states` |
+| `Text2Image` | text (and image) to image | any diffusers pipeline | `GeneratingModel` (`generate`) |
+| `Image2Mesh` — a base: subclass it with the model's `load` and `make_mesh` | image to mesh | the model the subclass loads, behind the task's `mesh` endpoint | `MeshingModel` (`mesh`: one picture in, a `GeneratedMesh` out) |
+| `TextRewriter` | text to text, rewritten | any encoder-decoder LM (T5, BART, Marian, ...) in the transformers layout, via `AutoModelForSeq2SeqLM` | `RewritingModel` (`rewrite`: one text in, one text out) |
+| `AnthropicText2Text` | text to text | the Anthropic API, forwarded from the cluster | `CompletingModel` (`complete`) |
+| `GeminiText2Text` | text to text | the Gemini API, forwarded from the cluster | `CompletingModel` (`complete`) |
+| `GeminiText2Image` | text to image | the Gemini API, forwarded from the cluster | `GeneratingModel` (`generate`) |
 
 | Importer | Source |
 |---|---|
@@ -85,11 +87,13 @@ need:
 | `.serve_app` | the class cortexgrid bundles and runs on the cluster |
 | `.requirements()` | what one replica needs to be placed and to run |
 | `.config()` | settings the serve app reads at construction, if it needs any |
-| `.client(url)` | a client that speaks the deployed app's routes |
 | `.source` | *(importers only)* a callable that downloads the weights and returns the directory |
 
 One with weights goes through `cortexgrid.import_model`, one without through
-`cortexgrid.register_model`.
+`cortexgrid.register_model`. Once deployed, the `cortexgrid.Deployment` hands back
+the client that calls the app's endpoints, typed as the serve app and so as its task
+type: `deployment.client()` once the app serves, or `deployment.client_async()` at
+once, with `await model.is_ready()` to ask.
 
 ## Quick start
 
@@ -117,13 +121,11 @@ cortexgrid.remote(
     import_weights, imp, imp.requirements(), num_gpus=0, num_cpus=2
 ).result()
 
-# 2. Deploy — blocks until the Ray Serve app is running.
-deployment = cortexgrid.deploy_model(
-    imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True, timeout=1800
-)
+# 2. Deploy.
+deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, timeout=1800)
 
-# 3. Inference.
-model = imp.client(deployment.url)
+# 3. Inference — the client blocks until the Ray Serve app is running.
+model: mg.GeneratingModel = deployment.client()
 
 async def run():
     result = await mg.generate(model, "a red bicycle on a beach at sunrise")
@@ -278,11 +280,13 @@ it is the floor; only quantizing the weights or running fewer forwards moves it.
 
 ## Inference
 
-`complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`):
+`complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`).
+`tools` are Python functions or tool specs; a function goes to the model as its spec,
+and each `ToolCall` comes back as data (`.id`, `.name`, `.arguments`) for the caller
+to run:
 
 ```python
-imp = mg.HuggingFaceImporter("Qwen/Qwen2.5-7B-Instruct", mg.Text2Text)
-model = imp.client(deployment.url)
+model: mg.CompletingModel = deployment.client()
 messages = [{"role": "user", "content": "Explain RAG in one sentence."}]
 
 async for chunk in mg.complete(model, messages, max_new_tokens=512, temperature=0.7):
@@ -323,9 +327,9 @@ a re-deploy rather than a re-registration.
 
 A replica asks for no hardware at all, so it is placed on any node, CPU-only
 included. From there it deploys and streams exactly like a cluster-served model:
-`entry.client(deployment.url)` returns the same `ServedCompletingModel`. Anthropic
+`deployment.client()` is a `CompletingModel` like `Text2Text`'s. Anthropic
 reports tool calls as structured blocks rather than as generated text, so the
-serve app re-encodes them into the text form the client parses.
+serve app turns them into `ToolCall`s as they come.
 
 ### Gemini
 
@@ -346,17 +350,16 @@ cortexgrid.register_model(
 
 Everything said above for Anthropic holds: `config` carries the model name and
 the name of the secret (`api_key_secret="..."` to use another), both editable on
-the model card; a replica needs no hardware; and `entry.client(deployment.url)`
-is a `ServedCompletingModel`, with Gemini's function calls re-encoded into the
-text form the client parses.
+the model card; a replica needs no hardware; and `deployment.client()`
+is a `CompletingModel`, with Gemini's function calls turned into `ToolCall`s.
 
-`GeminiText2Image` answers the same `/generate` route as `Text2Image`, so its
-client is a `ServedGeneratingModel`:
+`GeminiText2Image` is a `GeneratingModel` like `Text2Image`:
 
 ```python
 entry = mg.Hosted("gemini-2.5-flash-image", mg.GeminiText2Image)
 ...
-picture = await mg.generate(entry.client(deployment.url), "a fox in the snow", size=2048)
+model: mg.GeneratingModel = deployment.client()
+picture = await mg.generate(model, "a fox in the snow", size=2048)
 ```
 
 It takes a prompt only, and makes a square picture. `size` picks Gemini's size
@@ -368,40 +371,37 @@ diffusion settings with no Gemini counterpart, and come back as `None`.
 | | |
 |---|---|
 | `HuggingFaceImporter(hf_id, serve_app, token=None, ignore_patterns=None)` | Importer for a model on the HuggingFace Hub, run by `serve_app`. |
-| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` / `client(url)` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
+| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
 | `Hosted(model_id, serve_app, **settings)` | Entry for a model hosted elsewhere; no weights. `settings` are the serve app's `config` arguments. |
-| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`, `client(url)`. |
+| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`. |
 | `Text2Text`, `Text2Image`, `TextRewriter`, `Image2Mesh` | Serve apps that run weights (`LocalModel`s). |
 | `AnthropicText2Text` | Serve app that forwards to the Anthropic API (a `HostedModel`). |
 | `GeminiText2Text`, `GeminiText2Image` | Serve apps that forward to the Gemini API (`HostedModel`s sharing the `GeminiModel` base). |
-| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(url, name)`. |
-| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(url, name)`. |
+| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them). |
+| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`. |
 | `Weights(params=None, file_bytes=0)` | What an importer reads about the weights, handed to the serve app to size a replica. |
-| `complete(model, messages, tools=None, max_new_tokens=2048, temperature=0.7, **kw)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
-| `generate(model, prompt, *, image=None, **kw) -> GeneratedImage` | One image from a `GeneratingModel`. |
-| `ServedCompletingModel(url, model_id)` | Client for any deployed completion app. |
-| `ServedGeneratingModel(url, model_id)` | Client for any deployed text-to-image app. |
-| `ServedMeshingModel(url, model_id)` | Client for any deployed image-to-mesh app. |
+| `complete(model, messages, tools=None, max_new_tokens=None, temperature=0.7)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
+| `generate(model, prompt, *, image=None, steps=None, guidance=None, size=1024, seed=None) -> GeneratedImage` | One image from a `GeneratingModel`. |
 | `split_model_id(model_id) -> (family, suffix)` | The registry identity an importer derives. |
 | `detect_device()` | The torch device a serve app should load onto. |
 | `compiling.Mode.GRAPHED` / `compiling.Mode.FUSED` | The two modes a serve app chooses between. See [Compilation](#compilation). |
 | `compiling.supported(device)` / `compile_module(module, device, mode)` / `compile_pipeline(pipe, device, mode)` | Whether to compile here, and what a serve app calls to do it. |
 | `ModelDeployFailed` | Re-exported from cortexgrid; subclasses `RuntimeError`. |
 
-Types: `DeployedModel`, `CompletingModel`, `GeneratingModel`, `MeshingModel`,
+Types: `CompletingModel`, `GeneratingModel`, `MeshingModel`, `RewritingModel` (each
+the abstract base of its task's serve apps, declaring the task's endpoints),
 `CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
-`ToolSpec`.
+`ToolSpec`, `Tensor`.
 
 Extending it:
 
 - **A new source** is one importer: subclass `Importer` with `download(local_dir)`
   and `weights()`, and every serve app whose weight format it delivers runs its
   models unchanged.
-- **A new hosted API** is one serve app: subclass `HostedModel`, and register it
-  with `Hosted`. A text-to-text one only has to stream text from `POST /complete`,
-  inlining tool calls as `<tool_call>{"name": ..., "arguments": {...}}</tool_call>`
-  (there is an `encode_tool_call` for upstreams that report them structurally) —
-  then `ServedCompletingModel` is its client, unchanged.
+- **A new hosted API** is one serve app: subclass `HostedModel` and the task's type,
+  and register it with `Hosted`. A text-to-text one subclasses `CompletingModel` and
+  streams `CompletionChunk`s from its `complete` endpoint, with
+  `StreamedCompletionWriter` turning the upstream's text and tool calls into them.
 - **A model no task's serve app can load** gets its own serve app — an `Image2Mesh`
   subclass, say, with `load` and `make_mesh`, and `compile` if compiling it pays —
   and, if its source is unusual too, its own importer.

{cortexgrid_infer-0.3.7 → cortexgrid_infer-0.3.9}/README.md

@@ -22,23 +22,24 @@ Upgrading from 0.2.x? See [MIGRATION.md](https://github.com/robodatalab/cortexgr
 A model is two things, and the library keeps them apart:
 
 - **What runs it** — a serve app in `cortexgrid_infer.serve_apps`, named for the task
-  it runs, not for where the model came from. It loads the weights, answers the task's
-  routes (defined in `cortexgrid_infer.protocols`), and names the client that speaks
-  them. One serve app runs every model of its task, whichever source staged it.
+  it runs, not for where the model came from. It loads the weights and serves the
+  task's endpoints, declared by the task's type (`CompletingModel`, say); cortexgrid
+  generates the client that calls them. One serve app runs every model of its task,
+  whichever source staged it.
 - **Where its weights come from** — an importer in `cortexgrid_infer.importers`, one
   per source. It names the model in the registry, downloads the weights, and sizes
   them from the source's metadata. It is handed the serve app, so one importer takes
   any model its source holds.
 
-| Serve app | Task | Runs | Client |
+| Serve app | Task | Runs | Task type |
 |---|---|---|---|
-| `Text2Text` | text to text | any causal LM in the transformers layout, via `AutoModelForCausalLM` | `ServedCompletingModel` (`complete`) |
-| `Text2Image` | text (and image) to image | any diffusers pipeline | `ServedGeneratingModel` (`generate`) |
-| `Image2Mesh` — a base: subclass it with the model's `load` and `make_mesh` | image to mesh | the model the subclass loads, behind the task's `POST /mesh` | `ServedMeshingModel` (`mesh`: one picture in, a `GeneratedMesh` out) |
-| `TextRewriter` | text to text, rewritten | any encoder-decoder LM (T5, BART, Marian, ...) in the transformers layout, via `AutoModelForSeq2SeqLM` | `ServedRewritingModel` (`rewrite`: one text in, one text out) |
-| `AnthropicText2Text` | text to text | the Anthropic API, forwarded from the cluster | `ServedCompletingModel` (`complete`) |
-| `GeminiText2Text` | text to text | the Gemini API, forwarded from the cluster | `ServedCompletingModel` (`complete`) |
-| `GeminiText2Image` | text to image | the Gemini API, forwarded from the cluster | `ServedGeneratingModel` (`generate`) |
+| `Text2Text` | text to text | any causal LM in the transformers layout, via `AutoModelForCausalLM` | `CompletingModel` (`complete`), plus its own `loglikelihoods` and `last_hidden_states` |
+| `Text2Image` | text (and image) to image | any diffusers pipeline | `GeneratingModel` (`generate`) |
+| `Image2Mesh` — a base: subclass it with the model's `load` and `make_mesh` | image to mesh | the model the subclass loads, behind the task's `mesh` endpoint | `MeshingModel` (`mesh`: one picture in, a `GeneratedMesh` out) |
+| `TextRewriter` | text to text, rewritten | any encoder-decoder LM (T5, BART, Marian, ...) in the transformers layout, via `AutoModelForSeq2SeqLM` | `RewritingModel` (`rewrite`: one text in, one text out) |
+| `AnthropicText2Text` | text to text | the Anthropic API, forwarded from the cluster | `CompletingModel` (`complete`) |
+| `GeminiText2Text` | text to text | the Gemini API, forwarded from the cluster | `CompletingModel` (`complete`) |
+| `GeminiText2Image` | text to image | the Gemini API, forwarded from the cluster | `GeneratingModel` (`generate`) |
 
 | Importer | Source |
 |---|---|
@@ -57,11 +58,13 @@ need:
 | `.serve_app` | the class cortexgrid bundles and runs on the cluster |
 | `.requirements()` | what one replica needs to be placed and to run |
 | `.config()` | settings the serve app reads at construction, if it needs any |
-| `.client(url)` | a client that speaks the deployed app's routes |
 | `.source` | *(importers only)* a callable that downloads the weights and returns the directory |
 
 One with weights goes through `cortexgrid.import_model`, one without through
-`cortexgrid.register_model`.
+`cortexgrid.register_model`. Once deployed, the `cortexgrid.Deployment` hands back
+the client that calls the app's endpoints, typed as the serve app and so as its task
+type: `deployment.client()` once the app serves, or `deployment.client_async()` at
+once, with `await model.is_ready()` to ask.
 
 ## Quick start
 
@@ -89,13 +92,11 @@ cortexgrid.remote(
     import_weights, imp, imp.requirements(), num_gpus=0, num_cpus=2
 ).result()
 
-# 2. Deploy — blocks until the Ray Serve app is running.
-deployment = cortexgrid.deploy_model(
-    imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True, timeout=1800
-)
+# 2. Deploy.
+deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, timeout=1800)
 
-# 3. Inference.
-model = imp.client(deployment.url)
+# 3. Inference — the client blocks until the Ray Serve app is running.
+model: mg.GeneratingModel = deployment.client()
 
 async def run():
     result = await mg.generate(model, "a red bicycle on a beach at sunrise")
@@ -250,11 +251,13 @@ it is the floor; only quantizing the weights or running fewer forwards moves it.
 
 ## Inference
 
-`complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`):
+`complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`).
+`tools` are Python functions or tool specs; a function goes to the model as its spec,
+and each `ToolCall` comes back as data (`.id`, `.name`, `.arguments`) for the caller
+to run:
 
 ```python
-imp = mg.HuggingFaceImporter("Qwen/Qwen2.5-7B-Instruct", mg.Text2Text)
-model = imp.client(deployment.url)
+model: mg.CompletingModel = deployment.client()
 messages = [{"role": "user", "content": "Explain RAG in one sentence."}]
 
 async for chunk in mg.complete(model, messages, max_new_tokens=512, temperature=0.7):
@@ -295,9 +298,9 @@ a re-deploy rather than a re-registration.
 
 A replica asks for no hardware at all, so it is placed on any node, CPU-only
 included. From there it deploys and streams exactly like a cluster-served model:
-`entry.client(deployment.url)` returns the same `ServedCompletingModel`. Anthropic
+`deployment.client()` is a `CompletingModel` like `Text2Text`'s. Anthropic
 reports tool calls as structured blocks rather than as generated text, so the
-serve app re-encodes them into the text form the client parses.
+serve app turns them into `ToolCall`s as they come.
 
 ### Gemini
 
@@ -318,17 +321,16 @@ cortexgrid.register_model(
 
 Everything said above for Anthropic holds: `config` carries the model name and
 the name of the secret (`api_key_secret="..."` to use another), both editable on
-the model card; a replica needs no hardware; and `entry.client(deployment.url)`
-is a `ServedCompletingModel`, with Gemini's function calls re-encoded into the
-text form the client parses.
+the model card; a replica needs no hardware; and `deployment.client()`
+is a `CompletingModel`, with Gemini's function calls turned into `ToolCall`s.
 
-`GeminiText2Image` answers the same `/generate` route as `Text2Image`, so its
-client is a `ServedGeneratingModel`:
+`GeminiText2Image` is a `GeneratingModel` like `Text2Image`:
 
 ```python
 entry = mg.Hosted("gemini-2.5-flash-image", mg.GeminiText2Image)
 ...
-picture = await mg.generate(entry.client(deployment.url), "a fox in the snow", size=2048)
+model: mg.GeneratingModel = deployment.client()
+picture = await mg.generate(model, "a fox in the snow", size=2048)
 ```
 
 It takes a prompt only, and makes a square picture. `size` picks Gemini's size
@@ -340,40 +342,37 @@ diffusion settings with no Gemini counterpart, and come back as `None`.
 | | |
 |---|---|
 | `HuggingFaceImporter(hf_id, serve_app, token=None, ignore_patterns=None)` | Importer for a model on the HuggingFace Hub, run by `serve_app`. |
-| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` / `client(url)` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
+| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
 | `Hosted(model_id, serve_app, **settings)` | Entry for a model hosted elsewhere; no weights. `settings` are the serve app's `config` arguments. |
-| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`, `client(url)`. |
+| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`. |
 | `Text2Text`, `Text2Image`, `TextRewriter`, `Image2Mesh` | Serve apps that run weights (`LocalModel`s). |
 | `AnthropicText2Text` | Serve app that forwards to the Anthropic API (a `HostedModel`). |
 | `GeminiText2Text`, `GeminiText2Image` | Serve apps that forward to the Gemini API (`HostedModel`s sharing the `GeminiModel` base). |
-| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(url, name)`. |
-| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(url, name)`. |
+| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them). |
+| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`. |
 | `Weights(params=None, file_bytes=0)` | What an importer reads about the weights, handed to the serve app to size a replica. |
-| `complete(model, messages, tools=None, max_new_tokens=2048, temperature=0.7, **kw)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
-| `generate(model, prompt, *, image=None, **kw) -> GeneratedImage` | One image from a `GeneratingModel`. |
-| `ServedCompletingModel(url, model_id)` | Client for any deployed completion app. |
-| `ServedGeneratingModel(url, model_id)` | Client for any deployed text-to-image app. |
-| `ServedMeshingModel(url, model_id)` | Client for any deployed image-to-mesh app. |
+| `complete(model, messages, tools=None, max_new_tokens=None, temperature=0.7)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
+| `generate(model, prompt, *, image=None, steps=None, guidance=None, size=1024, seed=None) -> GeneratedImage` | One image from a `GeneratingModel`. |
 | `split_model_id(model_id) -> (family, suffix)` | The registry identity an importer derives. |
 | `detect_device()` | The torch device a serve app should load onto. |
 | `compiling.Mode.GRAPHED` / `compiling.Mode.FUSED` | The two modes a serve app chooses between. See [Compilation](#compilation). |
 | `compiling.supported(device)` / `compile_module(module, device, mode)` / `compile_pipeline(pipe, device, mode)` | Whether to compile here, and what a serve app calls to do it. |
 | `ModelDeployFailed` | Re-exported from cortexgrid; subclasses `RuntimeError`. |
 
-Types: `DeployedModel`, `CompletingModel`, `GeneratingModel`, `MeshingModel`,
+Types: `CompletingModel`, `GeneratingModel`, `MeshingModel`, `RewritingModel` (each
+the abstract base of its task's serve apps, declaring the task's endpoints),
 `CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
-`ToolSpec`.
+`ToolSpec`, `Tensor`.
 
 Extending it:
 
 - **A new source** is one importer: subclass `Importer` with `download(local_dir)`
   and `weights()`, and every serve app whose weight format it delivers runs its
   models unchanged.
-- **A new hosted API** is one serve app: subclass `HostedModel`, and register it
-  with `Hosted`. A text-to-text one only has to stream text from `POST /complete`,
-  inlining tool calls as `<tool_call>{"name": ..., "arguments": {...}}</tool_call>`
-  (there is an `encode_tool_call` for upstreams that report them structurally) —
-  then `ServedCompletingModel` is its client, unchanged.
+- **A new hosted API** is one serve app: subclass `HostedModel` and the task's type,
+  and register it with `Hosted`. A text-to-text one subclasses `CompletingModel` and
+  streams `CompletionChunk`s from its `complete` endpoint, with
+  `StreamedCompletionWriter` turning the upstream's text and tool calls into them.
 - **A model no task's serve app can load** gets its own serve app — an `Image2Mesh`
   subclass, say, with `load` and `make_mesh`, and `compile` if compiling it pays —
   and, if its source is unusual too, its own importer.

{cortexgrid_infer-0.3.7 → cortexgrid_infer-0.3.9}/cortexgrid_infer/__init__.py

@@ -7,9 +7,9 @@ nothing else, split along the two things a model is:
 
 - where its weights come from - an importer per source (`importers`), which
   names, fetches and sizes them;
-- what runs it - a serve app per task (`serve_apps`), which loads the weights,
-  answers the task's routes (`protocols`), and names the client that speaks
-  them.
+- what runs it - a serve app per task (`serve_apps`), which loads the weights
+  and serves the task's endpoints, declared by the task's type in `core`
+  (`CompletingModel`, say); cortexgrid generates the client that calls them.
 
     imp = cortexgrid_infer.HuggingFaceImporter("Qwen/Qwen2.5-0.5B-Instruct", cortexgrid_infer.Text2Text)
     with imp:
@@ -18,8 +18,8 @@ nothing else, split along the two things a model is:
             family=imp.family, suffix=imp.suffix,
             requirements=imp.requirements(),
         )
-    deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True)
-    model = imp.client(deployment.url)
+    deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED)
+    model: cortexgrid_infer.CompletingModel = deployment.client()
 
 A model hosted elsewhere has no weights and so no importer: its serve app
 forwards to it, and a `Hosted` entry registers it.
@@ -31,7 +31,6 @@ from cortexgrid import ModelDeployFailed
 
 from cortexgrid_infer.core import (
     CompletionChunk,
-    DeployedModel,
     CompletingModel,
     GeneratedImage,
     GeneratedMesh,
@@ -39,6 +38,7 @@ from cortexgrid_infer.core import (
     MeshingModel,
     Message,
     RewritingModel,
+    Tensor,
     Tool,
     ToolCall,
     ToolSpec,
@@ -47,12 +47,6 @@ from cortexgrid_infer.core import (
     mesh,
     rewrite,
 )
-from cortexgrid_infer.protocols import (
-    ServedCompletingModel,
-    ServedGeneratingModel,
-    ServedMeshingModel,
-    ServedRewritingModel,
-)
 from cortexgrid_infer.device import detect_device
 from cortexgrid_infer.serve_apps import (
     AnthropicText2Text,
@@ -71,7 +65,6 @@ from cortexgrid_infer.importers import HuggingFaceImporter, Importer
 
 __all__ = [
     "CompletionChunk",
-    "DeployedModel",
     "CompletingModel",
     "GeneratedImage",
     "GeneratedMesh",
@@ -79,6 +72,7 @@ __all__ = [
     "MeshingModel",
     "RewritingModel",
     "Message",
+    "Tensor",
     "Tool",
     "ToolCall",
     "ToolSpec",
@@ -88,10 +82,6 @@ __all__ = [
     "rewrite",
     "ModelDeployFailed",
     "detect_device",
-    "ServedCompletingModel",
-    "ServedGeneratingModel",
-    "ServedMeshingModel",
-    "ServedRewritingModel",
     "LocalModel",
     "HostedModel",
     "Weights",