nemo-platform-plugin 0.1.0__py3-none-any.whl

nemo_platform_plugin/.agents/skills/plugin-function/SKILL.md

@@ -0,0 +1,292 @@
+---
+name: plugin-function
+description: Creates in-process NemoFunction surfaces for NeMo Platform plugins. Use when adding a function, declaring spec_schema, mounting function routes with add_function_routes, understanding the two CLI verbs (run / submit), or streaming NDJSON frames. Trigger keywords - function, NemoFunction, spec_schema, add_function_routes, nemo_platform_plugin.functions, two verbs, run, submit, streaming, NDJSON, FunctionContext.
+---
+
+# Plugin Functions (NemoFunction)
+
+A `NemoFunction` is the third primitive on a plugin, alongside `NemoResource` and `NemoJob`. It's an in-process request handler — no scheduler, no backend dispatch — that the platform exposes as both a CLI subcommand and an HTTP route automatically.
+
+```text
+nemo <plugin> <fn> run    [--spec '{...}' | --spec-file FILE] [--workspace W] [<spec-flag>...]
+nemo <plugin> <fn> submit [--spec '{...}' | --spec-file FILE] \
+                          [--base-url URL | --cluster URL] \
+                          [--workspace W] [--request-id ID] [<spec-flag>...]
+```
+
+`run` is local (in-process); `submit` POSTs to the plugin service's auto-derived route. Two verbs only — no `explain`. A function's only schema is `spec_schema`, and `--help` is the introspection surface.
+
+## CLI introspection — auto-generated per-field flags
+
+Every scalar leaf in `spec_schema` becomes a Typer flag automatically. Nested submodels recurse with dotted paths (`--target.url`, `--target.timeout-seconds`). For a function with `spec_schema = GreetSpec(name: str)`:
+
+```text
+$ nemo my-plugin greet run --help
+...
+Function Spec:
+  --name <NAME>  Name to greet.
+
+Spec Source:
+  --spec <SPEC>            Spec as a JSON string. [default: {}]
+  --spec-file <SPEC_FILE>  Path to a YAML or JSON spec file (used as base; per-flag values override).
+  --workspace <WORKSPACE>  Workspace identity passed to the function as ctx.workspace. [default: default]
+...
+Function Spec flags are generated from the GreetSpec Pydantic schema. Precedence: --spec-file (base) → --spec JSON (overlay) → per-flag values (top).
+```
+
+Precedence at runtime is `--spec-file` (base) → `--spec` JSON (overlay) → per-field flag (top), then validation against `spec_schema` happens once on the merged value. Lists, dicts, discriminated unions, and other types the walker can't render fall through to `--spec` / `--spec-file`. The plumbing lives in `nemo_platform_plugin._spec_flags` and is shared with the jobs CLI; if a leaf is missing from `--help`, check that the field type is `str` / `int` / `float` / `bool` (or `Optional` of one of those) and that its name doesn't collide with a static verb flag.
+
+## Class Signature
+
+```python
+from typing import ClassVar
+from nemo_platform_plugin.function import NemoFunction
+from pydantic import BaseModel
+
+
+class GreetSpec(BaseModel):
+    name: str
+
+
+class GreetResponse(BaseModel):
+    message: str
+
+
+class GreetFunction(NemoFunction[GreetSpec]):
+    name:        ClassVar[str] = "greet"
+    description: ClassVar[str] = "Say hello to a name."
+    spec_schema: ClassVar[type[BaseModel]] = GreetSpec
+
+    async def run(self, spec: GreetSpec) -> GreetResponse:
+        return GreetResponse(message=f"Hello, {spec.name}!")
+```
+
+Required: `name`, `spec_schema`, `async def run()`. Optional: `description`, `endpoint`.
+
+## Method colours — `run` is **always** `async def`
+
+`NemoFunction` is single-coloured. `run` is the request handler, so it must be `async def` to avoid blocking the FastAPI event loop while a stream is open. The base class enforces this at class-definition time:
+
+```python
+class BadFunction(NemoFunction[GreetSpec]):
+    name = "bad"
+    spec_schema = GreetSpec
+
+    def run(self, spec):  # raises TypeError immediately at class definition
+        return {}
+```
+
+Both shapes are accepted:
+
+| Shape | What it returns | Wire content type |
+|---|---|---|
+| `async def run(self, spec)` | A value (typically a Pydantic model) | `application/json` |
+| `async def run(self, spec): yield ...` | An `AsyncIterator` of frames | `application/x-ndjson` |
+
+Pure sync work goes through `await asyncio.to_thread(...)`:
+
+```python
+async def run(self, spec: GreetSpec) -> GreetResponse:
+    rendered = await asyncio.to_thread(_blocking_template_render, spec.name)
+    return GreetResponse(message=rendered)
+```
+
+There is no `streaming: ClassVar[bool]` toggle — the route adapter detects the iterator return at request time.
+
+## Streaming
+
+Yield Pydantic models from an `async def` body:
+
+```python
+from collections.abc import AsyncIterator
+from nemo_platform_plugin.functions.frames import Done, Heartbeat
+from pydantic import BaseModel
+
+
+class TickSpec(BaseModel):
+    upto: int
+
+
+class Tick(BaseModel):
+    kind: str = "tick"
+    n: int
+
+
+class CountFunction(NemoFunction[TickSpec]):
+    name = "count"
+    spec_schema = TickSpec
+
+    async def run(self, spec: TickSpec) -> AsyncIterator[BaseModel]:
+        for i in range(spec.upto):
+            yield Tick(n=i)
+        yield Done()
+```
+
+Each yielded `BaseModel` becomes one NDJSON line (`model_dump_json() + "\n"`) on the wire. The route adapter wraps the iterator with a heartbeat injector that emits a `Heartbeat()` frame every 5 seconds of idle time so proxies don't time the connection out. Plugin authors don't need to emit heartbeats themselves — the framework adds them.
+
+Convention frames live in `nemo_platform_plugin.functions.frames`:
+
+| Class | `kind` | When to emit |
+|---|---|---|
+| `Heartbeat` | `"heartbeat"` | Framework injects on idle; plugins rarely emit explicitly |
+| `Done` | `"done"` | Last frame of a successful stream |
+| `Error` | `"error"` | Last frame of a failed stream |
+
+The discriminator is the `kind` field. Use it in your own frames so consumers can branch with `jq -r 'select(.kind == "tick") | .n'` etc.
+
+## Entry-Point Key Format
+
+Dot-separated: `"<plugin-name>.<function-name>"`.
+
+```toml
+[project.entry-points."nemo.functions"]
+"my-plugin.greet" = "nemo_my_plugin.functions.greet:GreetFunction"
+```
+
+`NemoFunction.name` is the **suffix** after the dot. For `"my-plugin.greet"` the class sets `name = "greet"`.
+
+## Mounting function routes — `add_function_routes(fn_cls)`
+
+This is the one-liner that mounts the function as a `POST` route on your plugin service:
+
+```python
+from fastapi import FastAPI
+from nemo_platform_plugin.functions.routes import add_function_routes
+
+from .functions.greet import GreetFunction
+
+app = FastAPI()
+router = add_function_routes(GreetFunction)
+app.include_router(
+    router,
+    prefix="/apis/my-plugin/v2/workspaces/{workspace}",
+)
+# POST /apis/my-plugin/v2/workspaces/{workspace}/greet
+```
+
+`add_function_routes`:
+
+- Validates the body against `spec_schema` (422 on bad input).
+- Builds a `FunctionContext` from the workspace path parameter and the optional `X-Request-ID` header.
+- Resolves keyword-only DI parameters on `run` by name — `ctx`, `async_sdk` (sync `sdk` injection lands in a follow-up).
+- Awaits `run(spec, **resolved)` and serialises the result; returns `application/x-ndjson` with heartbeat injection if the return is an async iterator.
+
+Optional kwarg `heartbeat_interval_seconds` overrides the 5-second default — useful in tests.
+
+## Endpoint override
+
+The default route path is `/{name}` mounted under the workspace prefix — i.e. `/apis/<plugin>/v2/workspaces/{workspace}/<name>`. Override the trailing segment per-class via `endpoint`:
+
+```python
+class GreetFunction(NemoFunction[GreetSpec]):
+    name = "greet"
+    spec_schema = GreetSpec
+    # Mount at /apis/.../v2/workspaces/{workspace}/greet/v1 to keep
+    # an existing client URL working.
+    endpoint: ClassVar[str | None] = "/{name}/v1"
+```
+
+Only `{name}` is substituted; the workspace placeholder stays as a live FastAPI route parameter. The override doesn't let a function relocate itself outside its plugin's URL namespace — that's intentional.
+
+## DI by signature
+
+`run` accepts framework-managed dependencies as keyword-only parameters. The route adapter and the local CLI both resolve them by parameter name:
+
+```python
+from nemo_platform_plugin.function import NemoFunction
+from nemo_platform_plugin.function_context import FunctionContext
+
+
+class WhoamiFunction(NemoFunction[GreetSpec]):
+    name = "whoami"
+    spec_schema = GreetSpec
+
+    async def run(
+        self,
+        spec: GreetSpec,
+        *,
+        ctx: FunctionContext,
+        async_sdk: object | None = None,
+    ) -> dict:
+        return {
+            "workspace": ctx.workspace,
+            "request_id": ctx.request_id,
+            "sdk_present": async_sdk is not None,
+        }
+```
+
+| Parameter | Source | Notes |
+|---|---|---|
+| `ctx: FunctionContext` | route adapter / CLI | `workspace` from URL or `--workspace`, `request_id` from `X-Request-ID` |
+| `async_sdk` | route adapter only (today) | Plugin services need `app.dependency_overrides[get_sdk_client]` to inject a real handle |
+| `sdk` | declared but currently bound to `None` | Sync placeholder lands with the SDK-builder follow-up |
+
+Parameters not declared on `run` aren't injected — there's no implicit context. Keep `run` signatures narrow.
+
+## Programmatic invocation
+
+```python
+import asyncio
+from nemo_platform_plugin.discovery import discover_functions
+from nemo_platform_plugin.function_context import FunctionContext
+
+fn_cls = discover_functions()["my-plugin.greet"]
+spec = fn_cls.spec_schema.model_validate({"name": "Ada"})
+result = asyncio.run(fn_cls().run(spec, ctx=FunctionContext(workspace="default")))
+```
+
+`discover_functions()` returns the typed mapping. There is no `FunctionScheduler` — the function is the runtime; instantiate and `await`.
+
+## Testing
+
+Unit-test functions directly — no FastAPI, no platform required:
+
+```python
+import pytest
+
+@pytest.mark.asyncio
+async def test_greet_returns_message():
+    fn = GreetFunction()
+    result = await fn.run(GreetSpec(name="Ada"))
+    assert result.message == "Hello, Ada!"
+```
+
+Route-test with `TestClient` when you want to exercise validation, DI, and streaming:
+
+```python
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from nemo_platform_plugin.functions.routes import add_function_routes
+
+
+def test_greet_route_returns_json():
+    app = FastAPI()
+    app.include_router(
+        add_function_routes(GreetFunction),
+        prefix="/apis/my-plugin/v2/workspaces/{workspace}",
+    )
+    client = TestClient(app)
+    response = client.post(
+        "/apis/my-plugin/v2/workspaces/default/greet",
+        json={"name": "Ada"},
+    )
+    assert response.status_code == 200
+    assert response.json() == {"message": "Hello, Ada!"}
+```
+
+For streaming functions, set a low heartbeat interval so tests aren't flaky:
+
+```python
+add_function_routes(CountFunction, heartbeat_interval_seconds=0)
+```
+
+## Gotchas
+
+- **`run` must be `async def`**: The class enforces this at definition time. Sync work uses `await asyncio.to_thread(...)`.
+- **`name` is the suffix only**: For entry-point key `"my-plugin.greet"`, `NemoFunction.name = "greet"`. The class declares the suffix; the dot-prefix lives in `pyproject.toml`.
+- **`spec_schema` is required**: `add_function_routes` raises `TypeError` if it's missing. Declare it before mounting.
+- **Streaming is detected from the return type, not declared**: An `async def` with `yield` is an async generator — the route emits NDJSON. An `async def` with `return` is a coroutine — the route emits JSON. Don't add a class-level streaming flag.
+- **`endpoint` is the trailing segment**: Setting `endpoint = "/{name}/v1"` mounts under the workspace prefix; you can't relocate the function to a different plugin's URL namespace.
+- **Reserved CLI flag**: `submit --workspace` controls the URL path segment — don't put a `workspace` field in your spec model with conflicting semantics.
+
+For the architectural framing alongside `NemoResource` and `NemoJob`, see the shared resources/jobs/functions design note.

nemo_platform_plugin/.agents/skills/plugin-inference-middleware/SKILL.md

@@ -0,0 +1,346 @@
+---
+name: plugin-inference-middleware
+description: Implements NemoInferenceMiddleware plugins for in-process inference request/response interception in IGW. Use when building a middleware plugin, implementing process_request or process_response, handling MiddlewareCall config (inline or config_id), exposing config entity CRUD APIs, or wiring up the nemo.inference_middleware entry-point. Trigger keywords: inference middleware, NemoInferenceMiddleware, process_request, process_response, MiddlewareCall, config_id, ImmediateResponse, VirtualModel middleware, middleware plugin.
+---
+
+# Inference Middleware Plugin
+
+## Entry-point registration
+
+```toml
+# pyproject.toml
+[project.entry-points."nemo.inference_middleware"]
+"nemo-my-plugin" = "nemo_my_plugin.middleware:MyMiddleware"
+```
+
+The key (`"nemo-my-plugin"`) is the **plugin identity** — it must match
+`MiddlewareCall.name` in VirtualModel configs.
+
+---
+
+## Minimal implementation
+
+```python
+from nemo_platform_plugin.inference_middleware import (
+    NemoInferenceMiddleware,
+    InferenceMiddlewareContext,
+    InferenceRequest,
+    InferenceResponse,
+    RequestResult,
+    ResponseResult,
+    ImmediateResponse,
+    InferenceMiddlewareError,
+    InferenceMiddlewareUnavailableError,
+)
+from nemo_platform_plugin.entity_client import NemoEntitiesClient
+from nmp.common.sdk_factory import get_async_platform_sdk
+from pydantic import BaseModel
+from typing import Any
+
+
+class MyConfigData(BaseModel):
+    """Lightweight working config — returned from validate_middleware_config."""
+    threshold: float = 0.8
+
+
+class MyMiddleware(NemoInferenceMiddleware):
+
+    def __init__(self) -> None:
+        super().__init__()
+        self._entity_client = None  # initialised in on_startup
+
+    # ── Lifecycle ──────────────────────────────────────────────────────
+    async def on_startup(self) -> None:
+        # Build an entity client from the platform SDK so get_middleware_config
+        # can fetch stored config entities. NemoEntitiesClient requires an
+        # AsyncEntitiesResource; obtain it via get_async_platform_sdk.
+        sdk = get_async_platform_sdk(as_service="nemo-my-plugin", internal=True)
+        self._entity_client = NemoEntitiesClient(sdk.entities)
+
+        entities = self.list_model_entities_for_workspace()  # cache available here
+        ...
+
+    async def on_shutdown(self) -> None: ...
+
+    async def on_virtual_model_upserted(self, virtual_model) -> None: ...
+    async def on_virtual_model_destroyed(self, virtual_model) -> None: ...
+
+    # ── Config resolution (needed only for config_id support) ──────────
+    async def get_middleware_config(self, config_type: str, config_id: str) -> Any:
+        """Fetch stored config entity from the entity store.
+
+        Called at VirtualModel cache-build time and every polling cycle.
+        Never called per-request.
+        """
+        if config_type != "my_plugin_config":
+            raise InferenceMiddlewareError(f"Unknown config_type={config_type!r}", status_code=400)
+        ws, name = config_id.split("/", 1)
+        return await self._entity_client.get(MyPluginConfig, name=name, workspace=ws)
+
+    async def validate_middleware_config(self, config_type: str, config: Any) -> MyConfigData:
+        """Validate and normalise config — called at cache-build time, not per-request."""
+        if config_type != "my_plugin_config":
+            raise ValueError(f"Unknown config_type={config_type!r}")
+        if isinstance(config, MyPluginConfig):
+            return MyConfigData(threshold=config.threshold)
+        return MyConfigData.model_validate(config)  # inline dict path
+
+    # ── Request hook ───────────────────────────────────────────────────
+    async def process_request(
+        self,
+        ctx: InferenceMiddlewareContext,
+        request: InferenceRequest,
+        middleware_config: MyConfigData,
+    ) -> RequestResult:
+        # Mutate body["model"] to route to a different entity:
+        request.body["model"] = "default/strong-model"
+        return request
+
+        # Or short-circuit (skip backend entirely):
+        # return ImmediateResponse(data={"choices": [{"message": {"content": "cached"}}], ...})
+
+        # Or raise a typed error:
+        # raise InferenceMiddlewareError("Rate limit exceeded", status_code=429)
+
+    # ── Response hook ──────────────────────────────────────────────────
+    async def process_response(
+        self,
+        ctx: InferenceMiddlewareContext,
+        response: InferenceResponse,
+        middleware_config: MyConfigData,
+    ) -> ResponseResult:
+        if isinstance(response.result, dict):
+            # Non-streaming — mutate and return
+            response.result["choices"][0]["message"]["content"] = redact(...)
+            return response.result
+        else:
+            # Streaming — wrap the async iterator
+            async def _filtered(stream):
+                async for chunk in stream:
+                    yield transform(chunk)
+            return _filtered(response.result)
+```
+
+---
+
+## MiddlewareCall: inline config vs. config_id
+
+```json
+// Inline config — embed directly in VirtualModel
+{
+  "name": "nemo-my-plugin",
+  "config_type": "my_plugin_config",
+  "config": {"threshold": 0.9}
+}
+
+// Config by reference — stored entity, supports versioning and sharing
+{
+  "name": "nemo-my-plugin",
+  "config_type": "my_plugin_config",
+  "config_id": "default/prod-config"
+}
+```
+
+`config_type` is **always required** regardless of which style is used.
+It must match the `entity_type` of your config `NemoEntity` subclass.
+
+**Use inline** when the config is simple, per-VirtualModel, and doesn't need
+versioning or sharing.
+
+**Use `config_id`** when:
+- Multiple VirtualModels share the same config
+- Operators update config independently of VirtualModels
+- Auto-propagation is needed — IGW re-resolves on every polling cycle when the
+  entity's `updated_at` changes; no VirtualModel edit required
+
+---
+
+## Config entity for config_id support
+
+When using `config_id`, the referenced entity must exist in the entity store.
+The plugin must expose CRUD endpoints for it:
+
+```python
+# entities.py
+from nemo_platform_plugin.entity import NemoEntity
+
+class MyPluginConfig(NemoEntity, entity_type="my_plugin_config"):
+    """Stored config — entity_type must match MiddlewareCall.config_type."""
+    threshold: float = 0.8
+    strong_model: str = "default/llama-70b"
+    weak_model: str = "default/llama-8b"
+```
+
+The CRUD API follows the standard NeMo Platform workspace-scoped pattern
+(`POST /v2/workspaces/{workspace}/my-plugin-configs`, etc.).
+See `plugin-service` skill for the full CRUD pattern.
+
+> **Separation of types:** `NemoEntity` inherits `workspace: str` (required,
+> no default) from `EntityBase`.  Inline config dicts (from `MiddlewareCall.config`)
+> contain only domain fields and have no `workspace`, so
+> `MyPluginConfig.model_validate({"threshold": 0.8})` raises a validation error.
+> A separate `MyConfigData(BaseModel)` with only the domain fields works for both
+> the inline path and the entity store path — have `validate_middleware_config`
+> always return that type.
+
+---
+
+## Cache accessor reference
+
+Available from `on_startup()` onward:
+
+```python
+# Model entities
+self.list_model_entities_for_workspace()               # all workspaces
+self.list_model_entities_for_workspace("default")     # filtered
+entity = self.get_model_entity("default/llama-3b")    # ModelEntity | None
+providers = self.get_model_providers_for_model("default/llama-3b")  # list[ModelProvider]
+
+# Resolve backend URL + served model name for a direct call
+target = self.get_inference_url_and_model("default/llama-3b")
+# target.model_provider_gateway_url  → "http://nim-svc:8080/v1"
+# target.served_model_name           → "meta/llama-3.2-3b-instruct"
+
+# VirtualModels
+vm = self.get_virtual_model("default/my-alias")       # VirtualModel | None
+self.list_virtual_models_for_workspace("default")     # list[str]
+```
+
+---
+
+## Request and response context
+
+### `request.typed_body`
+
+IGW populates `request.typed_body` with a TypedDict-validated view of the body
+for known paths (`v1/chat/completions`, `v1/messages`, `v1/responses`). All three
+SDK param types are TypedDicts — plain dicts at runtime. `typed_body` is
+interchangeable with `body`; use `request.path` for format dispatch, not
+`isinstance`.
+
+```python
+# Use typed_body when available, fall back to raw body
+body = request.typed_body if request.typed_body is not None else request.body
+
+# In a response hook, read the original pre-middleware request:
+original_body = ctx.original_request.typed_body  # or ctx.original_request.body
+```
+
+`ctx.original_request` is captured before any request middleware runs. Its
+`typed_body` and `.body` always reflect what the caller originally sent, even
+after downstream middleware has mutated the live request.
+
+### `response.typed_body` and annotations
+
+`response.typed_body` holds the SDK-native parsed response object
+(`ChatCompletion`, `Message`) when IGW can parse the backend payload. For
+non-streaming responses, if non-`None`, it is canonical — mutate it instead of
+`response.result`.
+
+Use this response contract:
+
+| Goal | How |
+|---|---|
+| Mutate an existing payload field (e.g. redact PII in `choices[0].message.content`) | Mutate `typed_body` when available, or `result` when no typed view exists |
+| Add a new field to the response body (e.g. `guardrails`) | Write to `response_body_annotations` |
+| Modify HTTP response headers | Mutate `headers` |
+
+```python
+# Add non-schema top-level metadata without fighting typed response serialization.
+response.response_body_annotations["guardrails_data"] = {
+    "config_ids": ["default/safety-config"]
+}
+return response
+```
+
+Request middleware can also annotate the eventual backend response, even though
+the `InferenceResponse` object does not exist yet. Put those annotations in
+`ctx.response_body_annotations` as a staging area:
+```python
+async def process_request(self, ctx, request, middleware_config):
+    ctx.response_body_annotations["guardrails_data"] = {
+        "config_ids": ["default/input-only-safety"]
+    }
+    return request
+```
+
+When IGW later receives the backend response, it builds an `InferenceResponse`
+and copies the staged values into `response.response_body_annotations`. From
+that point on, `response.response_body_annotations` is canonical because it is
+attached to the response being returned. Response middleware should preserve,
+replace, or remove annotations there, not on `ctx`. Final serialization injects
+only `response.response_body_annotations` into the response body.
+
+`response_body_annotations` is currently only supported for non-streaming responses.
+IGW accumulates annotations on streaming responses, but does not serialize
+them into the returned SSE chunks yet.
+
+---
+
+## RequestResult return values
+
+| Return | Effect |
+|---|---|
+| `InferenceRequest` | IGW resolves `request.body["model"]` to a provider and proxies |
+| `ImmediateResponse(data=...)` | Skip proxy; `data` is passed to response middleware |
+| Raise `InferenceMiddlewareError(msg, status_code=N)` | HTTP N error to caller |
+| Raise `InferenceMiddlewareUnavailableError(msg)` | HTTP 503 to caller |
+
+---
+
+## Testing pattern
+
+```python
+from unittest.mock import MagicMock
+from nemo_platform_plugin.inference_middleware import InferenceMiddlewareCacheAccessor
+
+def _make_plugin():
+    plugin = MyMiddleware()
+    cache = MagicMock(spec=InferenceMiddlewareCacheAccessor)
+    cache.list_model_entities_for_workspace.return_value = ["default/llama-3b"]
+    plugin._inject_cache(cache)
+    return plugin
+
+@pytest.mark.asyncio
+async def test_blocks_request():
+    plugin = _make_plugin()
+    cfg = MyConfigData(threshold=0.5)
+    result = await plugin.process_request({"model": "default/vm", "messages": []}, {}, cfg)
+    assert isinstance(result, ImmediateResponse)
+```
+
+To mock `get_middleware_config` (entity store fetch), patch at the entity client level:
+
+```python
+with patch("nemo_platform_plugin.entity_client.NemoEntitiesClient", return_value=mock_client):
+    result = await plugin.get_middleware_config("my_plugin_config", "ws/cfg")
+```
+
+---
+
+## Reference implementation
+
+`plugins/example-plugin/` contains a complete working example:
+
+| File | Contents |
+|---|---|
+| `middleware_config.py` | `ExampleMiddlewareConfig(NemoEntity)` — stored config entity |
+| `middleware.py` | `ExampleInferenceMiddleware` — full implementation with both config styles |
+| `middleware_service.py` | CRUD router for the config entity |
+| `tests/test_inference_middleware.py` | Unit tests with mocked cache and entity client |
+
+---
+
+## Common mistakes
+
+| Mistake | Fix |
+|---|---|
+| Calling `get_model_entity()` before `on_startup()` | Cache is only available after `_inject_cache()` |
+| Returning `None` from `process_request` | Always return the `InferenceRequest` or an `ImmediateResponse` |
+| Using old signature `process_request(self, request_body, request_headers, ...)` | Correct signature: `process_request(self, ctx, request, middleware_config)` |
+| Using old signature `process_response(self, response_result, request_body, ...)` | Correct signature: `process_response(self, ctx, response, middleware_config)` |
+| Accessing `request.body` via `request_body` argument | Use `request.body` — the parameter is now `request: InferenceRequest` |
+| Using `NemoEntity` directly as the `validate_middleware_config` return type | Return a lightweight `BaseModel` — entity metadata is irrelevant at request time |
+| Calling `get_middleware_config` per-request | This is called by IGW at cache-build time only; don't call it yourself in `process_request` |
+| Using `MiddlewareCall.config_id` without a CRUD API | The referenced entity must be creatable; expose CRUD endpoints |
+| Injecting new top-level response fields into `result` or `typed_body` | Use `response_body_annotations` so translation and typed serialization cannot drop them |