cortexgrid-infer 0.3.6__tar.gz → 0.3.8__tar.gz

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cortexgrid-infer
-Version: 0.3.6
+Version: 0.3.8
 Summary: Serve apps per task (text-to-text, text-to-image, image-to-mesh) and importers per source (HuggingFace) for running models on cortexgrid
 Project-URL: Homepage, https://github.com/robodatalab/cortexgrid-infer
 Project-URL: Repository, https://github.com/robodatalab/cortexgrid-infer
@@ -9,7 +9,7 @@ License-File: LICENSE
 Requires-Python: <3.12,>=3.11
 Requires-Dist: accelerate>=0.33
 Requires-Dist: anthropic>=0.40.0
-Requires-Dist: cortexgrid>=0.3.12
+Requires-Dist: cortexgrid>=0.3.16
 Requires-Dist: diffusers>=0.38
 Requires-Dist: fastapi>=0.133
 Requires-Dist: google-genai>=1.0
@@ -85,11 +85,12 @@ need:
 | `.serve_app` | the class cortexgrid bundles and runs on the cluster |
 | `.requirements()` | what one replica needs to be placed and to run |
 | `.config()` | settings the serve app reads at construction, if it needs any |
-| `.client(url)` | a client that speaks the deployed app's routes |
 | `.source` | *(importers only)* a callable that downloads the weights and returns the directory |
 
 One with weights goes through `cortexgrid.import_model`, one without through
-`cortexgrid.register_model`.
+`cortexgrid.register_model`. Once deployed, the `cortexgrid.Deployment` hands back
+the client that speaks the app's routes: `deployment.client()` once the app serves,
+or `deployment.client_async()` at once, with `await model.is_ready()` to ask.
 
 ## Quick start
 
@@ -117,13 +118,11 @@ cortexgrid.remote(
     import_weights, imp, imp.requirements(), num_gpus=0, num_cpus=2
 ).result()
 
-# 2. Deploy — blocks until the Ray Serve app is running.
-deployment = cortexgrid.deploy_model(
-    imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True, timeout=1800
-)
+# 2. Deploy.
+deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, timeout=1800)
 
-# 3. Inference.
-model = imp.client(deployment.url)
+# 3. Inference — the client blocks until the Ray Serve app is running.
+model = deployment.client()
 
 async def run():
     result = await mg.generate(model, "a red bicycle on a beach at sunrise")
@@ -281,8 +280,7 @@ it is the floor; only quantizing the weights or running fewer forwards moves it.
 `complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`):
 
 ```python
-imp = mg.HuggingFaceImporter("Qwen/Qwen2.5-7B-Instruct", mg.Text2Text)
-model = imp.client(deployment.url)
+model = deployment.client()
 messages = [{"role": "user", "content": "Explain RAG in one sentence."}]
 
 async for chunk in mg.complete(model, messages, max_new_tokens=512, temperature=0.7):
@@ -323,7 +321,7 @@ a re-deploy rather than a re-registration.
 
 A replica asks for no hardware at all, so it is placed on any node, CPU-only
 included. From there it deploys and streams exactly like a cluster-served model:
-`entry.client(deployment.url)` returns the same `ServedCompletingModel`. Anthropic
+`deployment.client()` returns the same `ServedCompletingModel`. Anthropic
 reports tool calls as structured blocks rather than as generated text, so the
 serve app re-encodes them into the text form the client parses.
 
@@ -346,7 +344,7 @@ cortexgrid.register_model(
 
 Everything said above for Anthropic holds: `config` carries the model name and
 the name of the secret (`api_key_secret="..."` to use another), both editable on
-the model card; a replica needs no hardware; and `entry.client(deployment.url)`
+the model card; a replica needs no hardware; and `deployment.client()`
 is a `ServedCompletingModel`, with Gemini's function calls re-encoded into the
 text form the client parses.
 
@@ -356,7 +354,8 @@ client is a `ServedGeneratingModel`:
 ```python
 entry = mg.Hosted("gemini-2.5-flash-image", mg.GeminiText2Image)
 ...
-picture = await mg.generate(entry.client(deployment.url), "a fox in the snow", size=2048)
+model = deployment.client()
+picture = await mg.generate(model, "a fox in the snow", size=2048)
 ```
 
 It takes a prompt only, and makes a square picture. `size` picks Gemini's size
@@ -368,28 +367,28 @@ diffusion settings with no Gemini counterpart, and come back as `None`.
 | | |
 |---|---|
 | `HuggingFaceImporter(hf_id, serve_app, token=None, ignore_patterns=None)` | Importer for a model on the HuggingFace Hub, run by `serve_app`. |
-| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` / `client(url)` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
+| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
 | `Hosted(model_id, serve_app, **settings)` | Entry for a model hosted elsewhere; no weights. `settings` are the serve app's `config` arguments. |
-| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`, `client(url)`. |
+| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`. |
 | `Text2Text`, `Text2Image`, `TextRewriter`, `Image2Mesh` | Serve apps that run weights (`LocalModel`s). |
 | `AnthropicText2Text` | Serve app that forwards to the Anthropic API (a `HostedModel`). |
 | `GeminiText2Text`, `GeminiText2Image` | Serve apps that forward to the Gemini API (`HostedModel`s sharing the `GeminiModel` base). |
-| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(url, name)`. |
-| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(url, name)`. |
+| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(deployment)`. |
+| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(deployment)`. |
 | `Weights(params=None, file_bytes=0)` | What an importer reads about the weights, handed to the serve app to size a replica. |
 | `complete(model, messages, tools=None, max_new_tokens=2048, temperature=0.7, **kw)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
 | `generate(model, prompt, *, image=None, **kw) -> GeneratedImage` | One image from a `GeneratingModel`. |
-| `ServedCompletingModel(url, model_id)` | Client for any deployed completion app. |
-| `ServedGeneratingModel(url, model_id)` | Client for any deployed text-to-image app. |
-| `ServedMeshingModel(url, model_id)` | Client for any deployed image-to-mesh app. |
+| `ServedCompletingModel(key, url)` | Client for any deployed completion app. |
+| `ServedGeneratingModel(key, url)` | Client for any deployed text-to-image app. |
+| `ServedMeshingModel(key, url)` | Client for any deployed image-to-mesh app. |
 | `split_model_id(model_id) -> (family, suffix)` | The registry identity an importer derives. |
 | `detect_device()` | The torch device a serve app should load onto. |
 | `compiling.Mode.GRAPHED` / `compiling.Mode.FUSED` | The two modes a serve app chooses between. See [Compilation](#compilation). |
 | `compiling.supported(device)` / `compile_module(module, device, mode)` / `compile_pipeline(pipe, device, mode)` | Whether to compile here, and what a serve app calls to do it. |
 | `ModelDeployFailed` | Re-exported from cortexgrid; subclasses `RuntimeError`. |
 
-Types: `DeployedModel`, `CompletingModel`, `GeneratingModel`, `MeshingModel`,
-`CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
+Types: `CompletingModel`, `GeneratingModel`, `MeshingModel`, `RewritingModel` (each
+a `cortexgrid.DeploymentClient`), `CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
 `ToolSpec`.
 
 Extending it:

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/README.md

@@ -57,11 +57,12 @@ need:
 | `.serve_app` | the class cortexgrid bundles and runs on the cluster |
 | `.requirements()` | what one replica needs to be placed and to run |
 | `.config()` | settings the serve app reads at construction, if it needs any |
-| `.client(url)` | a client that speaks the deployed app's routes |
 | `.source` | *(importers only)* a callable that downloads the weights and returns the directory |
 
 One with weights goes through `cortexgrid.import_model`, one without through
-`cortexgrid.register_model`.
+`cortexgrid.register_model`. Once deployed, the `cortexgrid.Deployment` hands back
+the client that speaks the app's routes: `deployment.client()` once the app serves,
+or `deployment.client_async()` at once, with `await model.is_ready()` to ask.
 
 ## Quick start
 
@@ -89,13 +90,11 @@ cortexgrid.remote(
     import_weights, imp, imp.requirements(), num_gpus=0, num_cpus=2
 ).result()
 
-# 2. Deploy — blocks until the Ray Serve app is running.
-deployment = cortexgrid.deploy_model(
-    imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True, timeout=1800
-)
+# 2. Deploy.
+deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, timeout=1800)
 
-# 3. Inference.
-model = imp.client(deployment.url)
+# 3. Inference — the client blocks until the Ray Serve app is running.
+model = deployment.client()
 
 async def run():
     result = await mg.generate(model, "a red bicycle on a beach at sunrise")
@@ -253,8 +252,7 @@ it is the floor; only quantizing the weights or running fewer forwards moves it.
 `complete` streams `CompletionChunk`s (`.content`, `.thinking`, `.tool_calls`, `.finish_reason`):
 
 ```python
-imp = mg.HuggingFaceImporter("Qwen/Qwen2.5-7B-Instruct", mg.Text2Text)
-model = imp.client(deployment.url)
+model = deployment.client()
 messages = [{"role": "user", "content": "Explain RAG in one sentence."}]
 
 async for chunk in mg.complete(model, messages, max_new_tokens=512, temperature=0.7):
@@ -295,7 +293,7 @@ a re-deploy rather than a re-registration.
 
 A replica asks for no hardware at all, so it is placed on any node, CPU-only
 included. From there it deploys and streams exactly like a cluster-served model:
-`entry.client(deployment.url)` returns the same `ServedCompletingModel`. Anthropic
+`deployment.client()` returns the same `ServedCompletingModel`. Anthropic
 reports tool calls as structured blocks rather than as generated text, so the
 serve app re-encodes them into the text form the client parses.
 
@@ -318,7 +316,7 @@ cortexgrid.register_model(
 
 Everything said above for Anthropic holds: `config` carries the model name and
 the name of the secret (`api_key_secret="..."` to use another), both editable on
-the model card; a replica needs no hardware; and `entry.client(deployment.url)`
+the model card; a replica needs no hardware; and `deployment.client()`
 is a `ServedCompletingModel`, with Gemini's function calls re-encoded into the
 text form the client parses.
 
@@ -328,7 +326,8 @@ client is a `ServedGeneratingModel`:
 ```python
 entry = mg.Hosted("gemini-2.5-flash-image", mg.GeminiText2Image)
 ...
-picture = await mg.generate(entry.client(deployment.url), "a fox in the snow", size=2048)
+model = deployment.client()
+picture = await mg.generate(model, "a fox in the snow", size=2048)
 ```
 
 It takes a prompt only, and makes a square picture. `size` picks Gemini's size
@@ -340,28 +339,28 @@ diffusion settings with no Gemini counterpart, and come back as `None`.
 | | |
 |---|---|
 | `HuggingFaceImporter(hf_id, serve_app, token=None, ignore_patterns=None)` | Importer for a model on the HuggingFace Hub, run by `serve_app`. |
-| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` / `client(url)` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
+| `Importer` | Base of the importers: identity, scratch directory, `source`, and `requirements()` taken from the serve app. A new source implements `download(local_dir)` and `weights()`. |
 | `Hosted(model_id, serve_app, **settings)` | Entry for a model hosted elsewhere; no weights. `settings` are the serve app's `config` arguments. |
-| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`, `client(url)`. |
+| `ModelEntry` | Base of both: `model_id`, `family`, `suffix`, `serve_app`, `requirements()`, `config()`. |
 | `Text2Text`, `Text2Image`, `TextRewriter`, `Image2Mesh` | Serve apps that run weights (`LocalModel`s). |
 | `AnthropicText2Text` | Serve app that forwards to the Anthropic API (a `HostedModel`). |
 | `GeminiText2Text`, `GeminiText2Image` | Serve apps that forward to the Gemini API (`HostedModel`s sharing the `GeminiModel` base). |
-| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(url, name)`. |
-| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(url, name)`. |
+| `LocalModel` | Base of the serve apps that run weights: `ignore_patterns()`, `bytes_per_param`, `min_vram_gb`, `requirements(weights)`, `config()` (the model card's defaults, `compile: "false"` among them), `client(deployment)`. |
+| `HostedModel` | Base of the serve apps that forward: `config(model_id, **settings)`, `requirements()`, `client(deployment)`. |
 | `Weights(params=None, file_bytes=0)` | What an importer reads about the weights, handed to the serve app to size a replica. |
 | `complete(model, messages, tools=None, max_new_tokens=2048, temperature=0.7, **kw)` | Async stream of `CompletionChunk` for a `CompletingModel`. |
 | `generate(model, prompt, *, image=None, **kw) -> GeneratedImage` | One image from a `GeneratingModel`. |
-| `ServedCompletingModel(url, model_id)` | Client for any deployed completion app. |
-| `ServedGeneratingModel(url, model_id)` | Client for any deployed text-to-image app. |
-| `ServedMeshingModel(url, model_id)` | Client for any deployed image-to-mesh app. |
+| `ServedCompletingModel(key, url)` | Client for any deployed completion app. |
+| `ServedGeneratingModel(key, url)` | Client for any deployed text-to-image app. |
+| `ServedMeshingModel(key, url)` | Client for any deployed image-to-mesh app. |
 | `split_model_id(model_id) -> (family, suffix)` | The registry identity an importer derives. |
 | `detect_device()` | The torch device a serve app should load onto. |
 | `compiling.Mode.GRAPHED` / `compiling.Mode.FUSED` | The two modes a serve app chooses between. See [Compilation](#compilation). |
 | `compiling.supported(device)` / `compile_module(module, device, mode)` / `compile_pipeline(pipe, device, mode)` | Whether to compile here, and what a serve app calls to do it. |
 | `ModelDeployFailed` | Re-exported from cortexgrid; subclasses `RuntimeError`. |
 
-Types: `DeployedModel`, `CompletingModel`, `GeneratingModel`, `MeshingModel`,
-`CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
+Types: `CompletingModel`, `GeneratingModel`, `MeshingModel`, `RewritingModel` (each
+a `cortexgrid.DeploymentClient`), `CompletionChunk`, `GeneratedImage`, `GeneratedMesh`, `ToolCall`, `Message`, `Tool`,
 `ToolSpec`.
 
 Extending it:

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/__init__.py

@@ -18,8 +18,8 @@ nothing else, split along the two things a model is:
             family=imp.family, suffix=imp.suffix,
             requirements=imp.requirements(),
         )
-    deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED, wait=True)
-    model = imp.client(deployment.url)
+    deployment = cortexgrid.deploy_model(imp.family, imp.suffix, cortexgrid.IMPORTED)
+    model = deployment.client()
 
 A model hosted elsewhere has no weights and so no importer: its serve app
 forwards to it, and a `Hosted` entry registers it.
@@ -31,7 +31,6 @@ from cortexgrid import ModelDeployFailed
 
 from cortexgrid_infer.core import (
     CompletionChunk,
-    DeployedModel,
     CompletingModel,
     GeneratedImage,
     GeneratedMesh,
@@ -71,7 +70,6 @@ from cortexgrid_infer.importers import HuggingFaceImporter, Importer
 
 __all__ = [
     "CompletionChunk",
-    "DeployedModel",
     "CompletingModel",
     "GeneratedImage",
     "GeneratedMesh",

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/core.py

@@ -8,6 +8,7 @@ from functools import partial
 from collections.abc import AsyncIterator
 from typing import Any, Callable, Sequence
 
+import cortexgrid
 import numpy as np
 
 
@@ -37,19 +38,34 @@ class CompletionChunk:
     def has_tool_calls(self) -> bool:
         return bool(self.tool_calls)
 
+    @classmethod
+    def from_wire(
+        cls, answered: dict[str, Any], tools_by_name: dict[str, Callable[..., Any]]
+    ) -> CompletionChunk:
+        tool_calls = []
+        for called in answered["tool_calls"]:
+            tool = tools_by_name.get(called["name"])
+            if tool is None:
+                continue
+            bound = partial(tool, **called["arguments"])
+            tool_call = ToolCall(
+                id=called["id"], name=called["name"], arguments=called["arguments"], _func=bound
+            )
+            tool_calls.append(tool_call)
+        return cls(
+            content=answered["content"],
+            thinking=answered["thinking"],
+            tool_calls=tool_calls,
+            finish_reason=answered["finish_reason"],
+        )
+
 
 Message = dict[str, Any]
 ToolSpec = dict[str, Any]
 Tool = Callable[..., Any] | ToolSpec
 
 
-class DeployedModel(abc.ABC):
-    @property
-    @abc.abstractmethod
-    def name(self) -> str: ...
-
-
-class CompletingModel(DeployedModel):
+class CompletingModel(cortexgrid.DeploymentClient, abc.ABC):
     @abc.abstractmethod
     def complete(
         self,
@@ -72,7 +88,7 @@ class GeneratedImage:
     params: dict[str, Any] = field(default_factory=dict)
 
 
-class GeneratingModel(DeployedModel):
+class GeneratingModel(cortexgrid.DeploymentClient, abc.ABC):
     """A deployed image model. Unlike completion, generation is a single
     request/response (no token streaming), so `generate` returns one result."""
 
@@ -103,14 +119,14 @@ class GeneratedMesh:
     params: dict[str, Any] = field(default_factory=dict)
 
 
-class MeshingModel(DeployedModel):
+class MeshingModel(cortexgrid.DeploymentClient, abc.ABC):
     """A deployed model that makes a mesh of the object in one picture."""
 
     @abc.abstractmethod
     async def mesh(self, image: bytes, **kwargs: Any) -> GeneratedMesh: ...
 
 
-class RewritingModel(DeployedModel):
+class RewritingModel(cortexgrid.DeploymentClient, abc.ABC):
     """A deployed model that rewrites one text into another - corrects,
     paraphrases, summarises or translates it. Like generation, a single
     request/response."""

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/importers/base.py

@@ -29,7 +29,6 @@ from typing import Self
 
 import cortexgrid
 
-from cortexgrid_infer.core import DeployedModel
 from cortexgrid_infer.registry import ModelEntry, split_model_id
 from cortexgrid_infer.serve_apps.base import LocalModel, Weights
 
@@ -97,6 +96,3 @@ class Importer(ModelEntry):
 
     def config(self) -> dict[str, str]:
         return self.serve_app.config()
-
-    def client(self, url: str) -> DeployedModel:
-        return self.serve_app.client(url, self.model_id)

cortexgrid_infer-0.3.8/cortexgrid_infer/protocols/completion.py

@@ -0,0 +1,83 @@
+"""The wire protocol between a cortexgrid-served completion app and its client.
+
+A completion serve app streams plain text from `POST /complete`, with tool calls
+inlined in the text as `<tool_call>{"name": ..., "arguments": {...}}</tool_call>`.
+That is the format an instruct-tuned causal LM emits on its own, so a serve app
+wrapping one forwards its tokens untouched; a serve app wrapping a provider with
+structured tool calls encodes them into the same form with `encode_tool_call`.
+
+Either way the client is the same `ServedCompletingModel`, which streams text
+back as `CompletionChunk`s and reassembles the tool calls.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+import json
+from typing import Any, Sequence
+
+import httpx
+
+from cortexgrid_infer.core import CompletingModel, CompletionChunk, Message, Tool
+from cortexgrid_infer.utils import build_tool_map, normalize_tools
+
+class ServedCompletingModel(CompletingModel):
+    """Client for a completion app cortexgrid has deployed at `url`.
+
+    Provider-agnostic: every completion serve app speaks the protocol above, so
+    which model is behind it only shows up in its deployment's `key`."""
+
+    async def loglikelihoods(
+        self, messages: list[Message], continuations: list[str]
+    ) -> list[float]:
+        body = {"messages": messages, "continuations": continuations}
+        loglikelihoods_url = f"{self.url}/loglikelihoods"
+        async with httpx.AsyncClient(timeout=None) as client:
+            response = await client.post(loglikelihoods_url, json=body)
+        response.raise_for_status()
+        answered = response.json()
+        return answered["loglikelihoods"]
+
+    async def complete(
+        self,
+        messages: list[Message],
+        tools: Sequence[Tool] | None = None,
+        max_new_tokens: int = 16 * 1024,
+        temperature: float = 0.7,
+        top_p: float = 0.9,
+        top_k: int = 50,
+        repetition_penalty: float = 1.0,
+        **kwargs: Any,
+    ) -> AsyncIterator[CompletionChunk]:
+        tool_specs = normalize_tools(tools)
+        tools_by_name = build_tool_map(tools)
+        body: dict[str, Any] = {
+            "messages": messages,
+            "tools": tool_specs,
+            "max_new_tokens": max_new_tokens,
+            "temperature": temperature,
+            "top_p": top_p,
+            "top_k": top_k,
+            "repetition_penalty": repetition_penalty,
+            **kwargs,
+        }
+        complete_url = f"{self.url}/complete"
+        async with httpx.AsyncClient(timeout=None) as client:
+            async with client.stream("POST", complete_url, json=body) as response:
+                response.raise_for_status()
+                async for line in response.aiter_lines():
+                    answered = json.loads(line)
+                    chunk = CompletionChunk.from_wire(answered, tools_by_name)
+                    yield chunk
+
+    async def last_hidden_states(
+        self, parts: list[str], continuations_of_each_part: list[list[str]]
+    ) -> AsyncIterator[list[list[float]]]:
+        body = {"parts": parts, "continuations_of_each_part": continuations_of_each_part}
+        last_hidden_states_url = f"{self.url}/last_hidden_states"
+        async with httpx.AsyncClient(timeout=None) as client:
+            async with client.stream("POST", last_hidden_states_url, json=body) as response:
+                response.raise_for_status()
+                async for line in response.aiter_lines():
+                    answered = json.loads(line)
+                    yield answered["last_hidden_states"]

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/protocols/imaging.py

@@ -15,7 +15,6 @@ task whoever made it.
 from __future__ import annotations
 
 import base64
-from dataclasses import dataclass
 from typing import Any
 
 import httpx
@@ -23,17 +22,9 @@ import httpx
 from cortexgrid_infer.core import GeneratedImage, GeneratingModel
 
 
-@dataclass
 class ServedGeneratingModel(GeneratingModel):
     """The client of any text-to-image serve app deployed at `url`."""
 
-    url: str
-    model_id: str
-
-    @property
-    def name(self) -> str:
-        return self.model_id
-
     async def generate(
         self,
         prompt: str,
@@ -69,6 +60,5 @@ class ServedGeneratingModel(GeneratingModel):
                 "guidance": data.get("guidance", guidance),
                 "seed": data.get("seed", seed),
                 "duration_s": data.get("duration_s"),
-                "model_id": self.model_id,
             },
         )

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/protocols/meshing.py

@@ -15,7 +15,6 @@ Every such app speaks it through
 from __future__ import annotations
 
 import base64
-from dataclasses import dataclass
 from typing import Any
 
 import httpx
@@ -41,17 +40,9 @@ def decode(data: str, kind: tuple[str, int]) -> np.ndarray:
     return np.frombuffer(base64.b64decode(data), dtype=dtype).reshape(-1, columns)
 
 
-@dataclass
 class ServedMeshingModel(MeshingModel):
     """The client of any image-to-mesh serve app deployed at `url`."""
 
-    url: str
-    model_id: str
-
-    @property
-    def name(self) -> str:
-        return self.model_id
-
     async def mesh(self, image: bytes, **options: Any) -> GeneratedMesh:
         body: dict[str, Any] = {"image": base64.b64encode(image).decode("ascii"), **options}
         # Meshing takes seconds; no client-side timeout.
@@ -67,6 +58,5 @@ class ServedMeshingModel(MeshingModel):
             params={
                 **data.get("params", {}),
                 "duration_s": data.get("duration_s"),
-                "model_id": self.model_id,
             },
         )

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/protocols/rewriting.py

@@ -15,7 +15,6 @@ every model of the task whoever made it.
 
 from __future__ import annotations
 
-from dataclasses import dataclass
 from typing import Any
 
 import httpx
@@ -23,17 +22,9 @@ import httpx
 from cortexgrid_infer.core import RewritingModel
 
 
-@dataclass
 class ServedRewritingModel(RewritingModel):
     """The client of any rewriting serve app deployed at `url`."""
 
-    url: str
-    model_id: str
-
-    @property
-    def name(self) -> str:
-        return self.model_id
-
     async def rewrite(
         self, text: str, max_new_tokens: int = 512, **generation_options: Any
     ) -> str:

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/registry.py

@@ -5,9 +5,9 @@ cortexgrid stores whatever weights it is handed under whatever key it is given,
 and serves them with whatever class it was told to bundle. It deliberately knows
 nothing about where a model came from or what shape it is. A `ModelEntry`
 supplies exactly that missing knowledge - the registry identity, the serve app,
-the hardware it needs, its config, and the client - and drives no lifecycle of
-its own: importing, deploying and deleting stay the caller's calls against the
-cortexgrid SDK.
+the hardware it needs, and its config - and drives no lifecycle of its own:
+importing, deploying and deleting stay the caller's calls against the cortexgrid
+SDK, and a deployment's client is the deployment's `client()`.
 
 A model with weights is an importer's (`cortexgrid_infer.importers`) and goes
 through `import_model`. A model hosted elsewhere is a `Hosted` entry and goes
@@ -27,7 +27,6 @@ import abc
 
 import cortexgrid
 
-from cortexgrid_infer.core import DeployedModel
 from cortexgrid_infer.serve_apps.base import HostedModel
 
 
@@ -49,8 +48,8 @@ def split_model_id(model_id: str) -> tuple[str, str]:
 class ModelEntry(abc.ABC):
     """One model as the registry files it and cortexgrid serves it."""
 
-    # The id the model is known by where it came from; the registry key and the
-    # client's name derive from it.
+    # The id the model is known by where it came from; the registry key derives
+    # from it.
     model_id: str
 
     # The class cortexgrid bundles and instantiates on the cluster.
@@ -72,10 +71,6 @@ class ModelEntry(abc.ABC):
         bundled code already knows everything it needs."""
         return {}
 
-    @abc.abstractmethod
-    def client(self, url: str) -> DeployedModel:
-        """Return a client for this model's serve app, deployed at `url`."""
-
 
 class Hosted(ModelEntry):
     """A model hosted elsewhere, served by a `HostedModel` app that forwards to it.
@@ -97,6 +92,3 @@ class Hosted(ModelEntry):
 
     def config(self) -> dict[str, str]:
         return dict(self._config)
-
-    def client(self, url: str) -> DeployedModel:
-        return self.serve_app.client(url, self.model_id)

{cortexgrid_infer-0.3.6 → cortexgrid_infer-0.3.8}/cortexgrid_infer/serve_apps/anthropic.py

@@ -19,12 +19,13 @@ from typing import Any
 import cortexgrid
 from anthropic import AsyncAnthropic
 from cortexgrid import serve
-from fastapi import FastAPI
+from fastapi import FastAPI, HTTPException, status
 from fastapi.responses import StreamingResponse
 
 from cortexgrid_infer.core import Message, ToolSpec
-from cortexgrid_infer.protocols.completion import ServedCompletingModel, encode_tool_call
+from cortexgrid_infer.protocols.completion import ServedCompletingModel
 from cortexgrid_infer.serve_apps.base import HostedModel
+from cortexgrid_infer.serve_apps.streamed_completion import StreamedCompletionWriter
 
 _app = FastAPI()
 
@@ -128,8 +129,8 @@ class AnthropicText2Text(HostedModel):
         return {MODEL_PARAM: model_id, API_KEY_SECRET_PARAM: api_key_secret}
 
     @classmethod
-    def client(cls, url: str, name: str) -> ServedCompletingModel:
-        return ServedCompletingModel(url=url, model_id=name)
+    def client(cls, deployment: cortexgrid.Deployment[ServedCompletingModel]) -> ServedCompletingModel:
+        return ServedCompletingModel(key=deployment.key, url=deployment.url)
 
     def __init__(self, deployment: cortexgrid.DeploymentKey) -> None:
         config = cortexgrid.model_config(deployment)
@@ -162,19 +163,30 @@ class AnthropicText2Text(HostedModel):
             create_kwargs["tools"] = tools
 
         async def stream() -> AsyncIterator[bytes]:
+            writer = StreamedCompletionWriter()
             async with self._client.messages.stream(**create_kwargs) as events:
                 async for event in events:
                     if (
                         event.type == "content_block_delta"
                         and event.delta.type == "text_delta"
                     ):
-                        yield event.delta.text.encode("utf-8")
+                        line = writer.read_content(event.delta.text)
+                        yield line
 
                     elif event.type == "content_block_stop":
                         block = events.current_message_snapshot.content[event.index]
                         if block.type == "tool_use":
-                            yield encode_tool_call(
-                                block.name, dict(block.input)
-                            ).encode("utf-8")
-
-        return StreamingResponse(stream(), media_type="text/plain")
+                            arguments = dict(block.input)
+                            line = writer.read_tool_call(block.name, arguments, block.id)
+                            yield line
+            for line in writer.finish():
+                yield line
+
+        return StreamingResponse(stream(), media_type="application/x-ndjson")
+
+    @_app.post("/last_hidden_states")
+    async def last_hidden_states(self, body: dict[str, Any]) -> None:
+        raise HTTPException(
+            status_code=status.HTTP_501_NOT_IMPLEMENTED,
+            detail="Anthropic does not expose its models' hidden states",
+        )