langchain-llama-cpp 0.1.0__tar.gz → 0.2.0__tar.gz

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langchain-llama-cpp
-Version: 0.1.0
+Version: 0.2.0
 Summary: An integration package connecting llama.cpp's llama-server and LangChain, with reasoning, timings, structured output and tool calling
 Keywords: langchain,llama.cpp,llama-server,llama-swap,llm,local-llm,chat-model
 Author: Bharathan Mudaliar
@@ -54,8 +54,12 @@ or `uv add langchain-llama-cpp`. Python 3.10+.
 
 - **llama-swap** in front of llama-server. `model` is the llama-swap entry to
   use. A bare llama-server (no llama-swap) is not tested yet.
-- llama-server build **b11429**, with gemma 4, Qwen3.6 and gpt-oss, on
-  langchain-core 1.6.
+- Live integration tests: gemma 4, thinking off and on. Last full run on
+  llama.cpp v0.9.0 (build b12110) with langchain-core 1.7.7 (2026-11-10).
+  Qwen3.6 and gpt-oss were tested on llama-server b11429 with langchain-core
+  1.6, and are no longer in the suite.
+- Request fields and reply shapes follow the llama.cpp source of the latest
+  release, **v0.9.0** (build b12110).
 
 ## Quick start
 
@@ -164,9 +168,35 @@ sees the schema's field descriptions: asked for an age described as "in months",
 it fills in years. Pass `include_format_instructions=True` with `json_schema` or
 `json_mode` to add the schema to the system message.
 
+### Model profile
+
+At construction, `profile` is filled from the server's `/props`
+(`/upstream/<model>/props` behind llama-swap, which starts the entry; `/props`
+on a plain llama-server):
+
+```python
+ChatLlama(model="gemma").profile
+# {'text_inputs': True, 'max_input_tokens': 262144, 'image_inputs': False,
+#  'audio_inputs': False, 'video_inputs': False, 'tool_calling': True}
+```
+
+`max_input_tokens` is the per-slot context window (`n_ctx`), which LangChain and
+Deep Agents use to size things such as summarization. It's fetched once per
+`(base_url, model)` per process. If the server can't be reached, `profile` stays
+`None`. Pass `profile=` to set your own, or `fetch_profile=False` to skip the
+request. The request uses a separate client, not your `http_client`, which is
+used only for chat requests.
+
+`structured_output` is deliberately not set, so `create_agent` uses
+`ToolStrategy` for a `response_format`. llama-server can't combine `tools` with
+a `json_schema` `response_format`. The response-format grammar replaces the
+tool-call grammar, so the model answers in JSON straight away and never calls a
+tool. With Qwen-style templates in v0.9.0 the request fails instead, with
+"failed to parse grammar".
+
 ### Request parameters
 
-Every field `/v1/chat/completions` accepts in b11429 is a typed parameter:
+Every field `/v1/chat/completions` accepts in v0.9.0 is a typed parameter:
 `temperature`, `top_k`, `top_p`, `min_p`, `samplers`, `repeat_penalty`, the DRY
 and XTC samplers, `grammar`, `json_schema`, `reasoning_budget_tokens`,
 `cache_prompt`, `id_slot`, `lora` and more. Set them on the model or per call:
@@ -213,13 +243,14 @@ seconds and defaults to `None` (no timeout), since local generation can be slow.
   models lets the model write text before the call and won't let it stop
   without one. A model that doesn't want a tool keeps writing until
   `max_tokens`, and there's no tool call. This is
-  [ggml-org/llama.cpp#27217](https://github.com/ggml-org/llama.cpp/issues/27217);
-  keep `max_tokens` set.
+  [ggml-org/llama.cpp#27217](https://github.com/ggml-org/llama.cpp/issues/27217)
+  (closed as stale without a fix); keep `max_tokens` set.
 - **A named `tool_choice`** is sent as `"required"` (see Tool calling).
 - **JSON schema bounds on numbers** (`minimum`, `maximum`, `multipleOf` on
   `number`) aren't enforced by llama-server's grammar. Integer bounds are.
 - **Bare llama-server** (without llama-swap) isn't tested; `get_server_defaults`
-  needs llama-swap's `/upstream/<model>/props`.
+  needs llama-swap's `/upstream/<model>/props` (the profile falls back to `/props`).
+- **Tools and `response_format` together** give no tool calls (see Model profile).
 
 ## Development
 
@@ -232,9 +263,9 @@ uv run pytest tests/unit_tests          # offline
 uv run ruff check && uv run ruff format --check && uv run pyright
 ```
 
-The integration tests (LangChain's standard `langchain-tests` suite, on gemma,
-qwen and gpt-oss) need a llama-swap server. Copy `.env.example` to `.env` and
-set `LLAMA_BASE_URL`, then:
+The integration tests (LangChain's standard `langchain-tests` suite, on Gemma 4
+with thinking off and on) need a llama-swap server with a `gemma` entry. Copy
+`.env.example` to `.env` and set `LLAMA_BASE_URL`, then:
 
 ```bash
 uv run --env-file .env pytest tests/integration_tests

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/README.md

@@ -28,8 +28,12 @@ or `uv add langchain-llama-cpp`. Python 3.10+.
 
 - **llama-swap** in front of llama-server. `model` is the llama-swap entry to
   use. A bare llama-server (no llama-swap) is not tested yet.
-- llama-server build **b11429**, with gemma 4, Qwen3.6 and gpt-oss, on
-  langchain-core 1.6.
+- Live integration tests: gemma 4, thinking off and on. Last full run on
+  llama.cpp v0.9.0 (build b12110) with langchain-core 1.7.7 (2026-11-10).
+  Qwen3.6 and gpt-oss were tested on llama-server b11429 with langchain-core
+  1.6, and are no longer in the suite.
+- Request fields and reply shapes follow the llama.cpp source of the latest
+  release, **v0.9.0** (build b12110).
 
 ## Quick start
 
@@ -138,9 +142,35 @@ sees the schema's field descriptions: asked for an age described as "in months",
 it fills in years. Pass `include_format_instructions=True` with `json_schema` or
 `json_mode` to add the schema to the system message.
 
+### Model profile
+
+At construction, `profile` is filled from the server's `/props`
+(`/upstream/<model>/props` behind llama-swap, which starts the entry; `/props`
+on a plain llama-server):
+
+```python
+ChatLlama(model="gemma").profile
+# {'text_inputs': True, 'max_input_tokens': 262144, 'image_inputs': False,
+#  'audio_inputs': False, 'video_inputs': False, 'tool_calling': True}
+```
+
+`max_input_tokens` is the per-slot context window (`n_ctx`), which LangChain and
+Deep Agents use to size things such as summarization. It's fetched once per
+`(base_url, model)` per process. If the server can't be reached, `profile` stays
+`None`. Pass `profile=` to set your own, or `fetch_profile=False` to skip the
+request. The request uses a separate client, not your `http_client`, which is
+used only for chat requests.
+
+`structured_output` is deliberately not set, so `create_agent` uses
+`ToolStrategy` for a `response_format`. llama-server can't combine `tools` with
+a `json_schema` `response_format`. The response-format grammar replaces the
+tool-call grammar, so the model answers in JSON straight away and never calls a
+tool. With Qwen-style templates in v0.9.0 the request fails instead, with
+"failed to parse grammar".
+
 ### Request parameters
 
-Every field `/v1/chat/completions` accepts in b11429 is a typed parameter:
+Every field `/v1/chat/completions` accepts in v0.9.0 is a typed parameter:
 `temperature`, `top_k`, `top_p`, `min_p`, `samplers`, `repeat_penalty`, the DRY
 and XTC samplers, `grammar`, `json_schema`, `reasoning_budget_tokens`,
 `cache_prompt`, `id_slot`, `lora` and more. Set them on the model or per call:
@@ -187,13 +217,14 @@ seconds and defaults to `None` (no timeout), since local generation can be slow.
   models lets the model write text before the call and won't let it stop
   without one. A model that doesn't want a tool keeps writing until
   `max_tokens`, and there's no tool call. This is
-  [ggml-org/llama.cpp#27217](https://github.com/ggml-org/llama.cpp/issues/27217);
-  keep `max_tokens` set.
+  [ggml-org/llama.cpp#27217](https://github.com/ggml-org/llama.cpp/issues/27217)
+  (closed as stale without a fix); keep `max_tokens` set.
 - **A named `tool_choice`** is sent as `"required"` (see Tool calling).
 - **JSON schema bounds on numbers** (`minimum`, `maximum`, `multipleOf` on
   `number`) aren't enforced by llama-server's grammar. Integer bounds are.
 - **Bare llama-server** (without llama-swap) isn't tested; `get_server_defaults`
-  needs llama-swap's `/upstream/<model>/props`.
+  needs llama-swap's `/upstream/<model>/props` (the profile falls back to `/props`).
+- **Tools and `response_format` together** give no tool calls (see Model profile).
 
 ## Development
 
@@ -206,9 +237,9 @@ uv run pytest tests/unit_tests          # offline
 uv run ruff check && uv run ruff format --check && uv run pyright
 ```
 
-The integration tests (LangChain's standard `langchain-tests` suite, on gemma,
-qwen and gpt-oss) need a llama-swap server. Copy `.env.example` to `.env` and
-set `LLAMA_BASE_URL`, then:
+The integration tests (LangChain's standard `langchain-tests` suite, on Gemma 4
+with thinking off and on) need a llama-swap server with a `gemma` entry. Copy
+`.env.example` to `.env` and set `LLAMA_BASE_URL`, then:
 
 ```bash
 uv run --env-file .env pytest tests/integration_tests

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "langchain-llama-cpp"
-version = "0.1.0"
+version = "0.2.0"
 description = "An integration package connecting llama.cpp's llama-server and LangChain, with reasoning, timings, structured output and tool calling"
 readme = "README.md"
 license = "MIT"

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "langchain-llama-cpp"
-version = "0.1.0"
+version = "0.2.0"
 description = "An integration package connecting llama.cpp's llama-server and LangChain, with reasoning, timings, structured output and tool calling"
 readme = "README.md"
 license = "MIT"

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/src/langchain_llama_cpp/_wire.py

@@ -1,4 +1,4 @@
-"""The JSON llama-server sends back from `/v1/chat/completions` (b11429).
+"""The JSON llama-server sends back from `/v1/chat/completions` (v0.9.0).
 
 Types for the type checker only: replies are plain dicts from `json.loads`,
 never validated, so a server that sends something else isn't rejected here.
@@ -6,7 +6,7 @@ Every class is `total=False` because the code reads each key with `.get()`
 and copes when it's missing; the docstrings say which keys the server always
 sends.
 
-Sources (llama.cpp b11429): `tools/server/server-task.cpp`
+Sources (llama.cpp v0.9.0, build b12110): `tools/server/server-task.cpp`
 (`to_json_oaicompat_chat`, `to_json_oaicompat_chat_stream`,
 `server_task_result_cmpl_partial::to_json_oaicompat_chat`,
 `probs_vector_to_json`, `result_prompt_progress::to_json`),
@@ -55,7 +55,7 @@ the stream ends without `[DONE]`.
 from typing import Literal, TypedDict
 
 FinishReason = Literal["stop", "length", "tool_calls"]
-"""`length` when `max_tokens` or the context window ran out (b11429 sends
+"""`length` when `max_tokens` or the context window ran out (v0.9.0 sends
 `length` for every stop that isn't an end-of-sequence token or a stop string)."""
 
 
@@ -101,6 +101,9 @@ class LlamaContentPart(TypedDict, total=False):
     type: str
     text: str
 
+    name: str
+    """In place of `text` on a `tool_reference` part."""
+
 
 class LlamaMessage(TypedDict, total=False):
     """`choices[].message` in a reply. Always `role` and `content`."""
@@ -190,7 +193,7 @@ class LlamaPromptTokensDetails(TypedDict, total=False):
 class LlamaCompletionTokensDetails(TypedDict, total=False):
     """`usage.completion_tokens_details`, as OpenAI sends it.
 
-    b11429 never sends it (reasoning tokens are counted in
+    v0.9.0 never sends it (reasoning tokens are counted in
     `completion_tokens` but not reported apart); read in case a build does.
     """
 
@@ -198,7 +201,7 @@ class LlamaCompletionTokensDetails(TypedDict, total=False):
 
 
 class LlamaUsage(TypedDict, total=False):
-    """`usage`. b11429 always sends the four keys below except the last."""
+    """`usage`. v0.9.0 always sends the four keys below except the last."""
 
     prompt_tokens: int
 

{langchain_llama_cpp-0.1.0 → langchain_llama_cpp-0.2.0}/src/langchain_llama_cpp/chat_models.py

@@ -20,6 +20,7 @@ from langchain_core.language_models import (
     LangSmithParams,
     LanguageModelInput,
 )
+from langchain_core.language_models.model_profile import ModelProfile
 from langchain_core.messages import (
     AIMessage,
     AIMessageChunk,
@@ -94,6 +95,10 @@ INITIAL_RETRY_DELAY = 0.5
 MAX_RETRY_DELAY = 8.0
 MAX_RETRY_AFTER = 60.0
 
+# The `/props` fetch behind `profile`. Behind llama-swap it starts the entry,
+# which can take tens of seconds for a large model; connecting must not.
+PROFILE_TIMEOUT = httpx.Timeout(120.0, connect=10.0)
+
 # ChatLlama fields that configure the client rather than the request.
 _CLIENT_FIELDS = frozenset(
     {
@@ -103,11 +108,16 @@ _CLIENT_FIELDS = frozenset(
         "max_retries",
         "streaming",
         "stream_usage",
+        "fetch_profile",
         "http_client",
         "http_async_client",
     }
 )
 
+# Profiles fetched from `/props`, per (base_url, model), for the process.
+# Only successes are kept, so a server that was down is asked again.
+_PROFILE_CACHE: dict[tuple[str, str], ModelProfile] = {}
+
 _AsyncClients = weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, httpx.AsyncClient]
 
 # What `_chunk_from_line` returns for `data: [DONE]`, the end of a finished stream.
@@ -182,7 +192,7 @@ class ChatLlama(BaseChatModel):
     """Server root, e.g. `http://localhost:8080`. Defaults to `LLAMA_BASE_URL`."""
 
     # --- Request fields -----------------------------------------------------
-    # Every field accepted by /v1/chat/completions in llama.cpp b11429
+    # Every field accepted by /v1/chat/completions in llama.cpp v0.9.0 (b12110)
     # (tools/server/server-common.cpp oaicompat_chat_params_parse,
     # server-schema.cpp make_llama_cmpl_schema, server-context.cpp id_slot).
     # Each is sent under its own name; None (the default) leaves it out, so the
@@ -338,7 +348,11 @@ class ChatLlama(BaseChatModel):
     """Continue the last assistant message instead of starting a new turn."""
 
     echo: bool | None = None
-    """Echo the input tokens in the output."""
+    """With `continue_final_message`, include the continued text in the reply.
+
+    By default the reply holds only the newly generated text. (In b11429 and
+    older builds it meant echoing the input tokens.)
+    """
 
     # Probabilities and debugging output
     logprobs: bool | None = None
@@ -406,8 +420,23 @@ class ChatLlama(BaseChatModel):
     `timings`. Can be overridden per call with `stream_usage=`.
     """
 
+    fetch_profile: bool = True
+    """Fill `profile` from the server's `/props` at construction (default on).
+
+    Reads `GET /upstream/<model>/props` (llama-swap; this starts the entry),
+    falling back to `GET /props` (a plain llama-server) on a 404. Gives
+    `max_input_tokens` (the per-slot `n_ctx`), the input modalities and
+    `tool_calling`. Fetched once per `(base_url, model)` per process. Any failure
+    leaves `profile` at `None`. A `profile=` you pass is used as it is, with no
+    fetch. Turn it off for offline use. `/props` is fetched with a separate
+    client, not `http_client`.
+    """
+
     http_client: httpx.Client | None = Field(default=None, exclude=True)
-    """Your own `httpx.Client`. If given, `timeout` is not applied to it."""
+    """Your own `httpx.Client` for chat requests (not the `/props` profile fetch).
+
+    If given, `timeout` is not applied to it.
+    """
 
     http_async_client: httpx.AsyncClient | None = Field(default=None, exclude=True)
     """Your own `httpx.AsyncClient` for async calls. `timeout` isn't applied to it."""
@@ -495,8 +524,9 @@ class ChatLlama(BaseChatModel):
                 (the handler llama.cpp names "qwen3-coder") the grammar lets
                 the model write text before the call and forbids ending without
                 one, so a model that doesn't want a tool keeps writing until
-                `max_tokens`, with no tool call (llama.cpp issue #27217, open
-                as of b11429). Keep `max_tokens` set when forcing tools.
+                `max_tokens`, with no tool call (llama.cpp issue #27217, closed
+                as stale without a fix; unchanged as of v0.9.0). Keep
+                `max_tokens` set when forcing tools.
             strict: Sent as `strict` on each tool when set. llama-server
                 ignores it: its grammar always enforces the schema.
             parallel_tool_calls: Sent when set.
@@ -521,7 +551,7 @@ class ChatLlama(BaseChatModel):
             ]
 
             # llama-server takes only these strings. It turns the object form
-            # into "auto" (b11429), so a named tool becomes "required".
+            # into "auto" (v0.9.0), so a named tool becomes "required".
             if tool_choice is True or tool_choice == "any":
                 tool_choice = "required"
             elif not (
@@ -718,6 +748,52 @@ class ChatLlama(BaseChatModel):
 
     # --- LangChain core hooks ---------------------------------------------
 
+    def _resolve_model_profile(self) -> ModelProfile | None:
+        """The profile from the server's `/props`, or None (see `fetch_profile`).
+
+        Core calls this at construction when no `profile` was passed, and turns
+        any exception into None; failures return None here already.
+        """
+        if not self.fetch_profile:
+            return None
+
+        base_url = self.base_url.rstrip("/")
+        key = (base_url, self.model)
+        if (cached := _PROFILE_CACHE.get(key)) is None:
+            props = self._fetch_props(base_url)
+            profile = _profile_from_props(props) if props is not None else None
+            if profile is None:
+                return None
+            _PROFILE_CACHE[key] = cached = profile
+        return ModelProfile(**cached)  # a copy: the caller may change it
+
+    def _fetch_props(self, base_url: str) -> dict[str, Any] | None:
+        """`/props` of this model's server: llama-swap's path, then llama-server's.
+
+        One attempt each, no retries, with `PROFILE_TIMEOUT`. Always on a
+        short-lived client of its own, never `http_client`: that client's event
+        hooks (e.g. a recorder of chat requests) shouldn't see this GET.
+        ChatLlama sends no auth headers, so nothing from it is needed.
+        """
+        paths = (f"/upstream/{self.model}/props", "/props")
+        client = _props_client()
+        try:
+            for path in paths:
+                response = client.get(base_url + path, timeout=PROFILE_TIMEOUT)
+                if response.status_code == 404:  # noqa: PLR2004
+                    continue  # not llama-swap: try a plain llama-server
+                if not response.is_success:
+                    return None
+                props: object = response.json()
+                return (
+                    cast("dict[str, Any]", props) if isinstance(props, dict) else None
+                )
+        except (httpx.HTTPError, ValueError):  # unreachable, timed out, not JSON
+            return None
+        finally:
+            client.close()
+        return None
+
     def _get_ls_params(
         self, stop: list[str] | None = None, **kwargs: Any
     ) -> LangSmithParams:
@@ -1040,6 +1116,60 @@ def _without_none(**items: Any) -> dict[str, Any]:
     return {key: value for key, value in items.items() if value is not None}
 
 
+# --- Profile --------------------------------------------------------------
+
+
+def _props_client() -> httpx.Client:
+    """A new client for the `/props` fetch (a function so tests can swap it)."""
+    return httpx.Client()
+
+
+def _profile_from_props(props: Mapping[str, Any]) -> ModelProfile | None:
+    """A `ModelProfile` from llama-server's `/props`, or None if it isn't one.
+
+    Fields as of llama.cpp v0.9.0 (`server-context.cpp` `get_res_props`):
+
+    - `default_generation_settings.n_ctx`: the per-slot context window, which
+      holds the prompt and the reply → `max_input_tokens`.
+    - `modalities.{vision, audio, video}` → `image_inputs`, `audio_inputs`,
+      `video_inputs`.
+    - `chat_template_caps.supports_tool_calls` → `tool_calling`. Without it the
+      server's generic parser doesn't parse tool calls at all
+      (`chat-auto-parser-generator.cpp`).
+
+    `structured_output` is left out on purpose: with `tools` in the request, a
+    `response_format` grammar replaces the tool-call grammar in every chat
+    handler, so an agent's model could no longer call tools.
+    """
+    settings: object = props.get("default_generation_settings")
+    if not isinstance(settings, dict):
+        return None  # a 200 from something that isn't llama-server
+
+    profile = ModelProfile(text_inputs=True)
+
+    n_ctx: object = cast("dict[str, Any]", settings).get("n_ctx")
+    if isinstance(n_ctx, int) and not isinstance(n_ctx, bool) and n_ctx > 0:
+        profile["max_input_tokens"] = n_ctx
+
+    modalities: object = props.get("modalities")
+    if isinstance(modalities, dict):
+        modality = cast("dict[str, Any]", modalities)
+        if isinstance(vision := modality.get("vision"), bool):
+            profile["image_inputs"] = vision
+        if isinstance(audio := modality.get("audio"), bool):
+            profile["audio_inputs"] = audio
+        if isinstance(video := modality.get("video"), bool):
+            profile["video_inputs"] = video
+
+    caps: object = props.get("chat_template_caps")
+    if isinstance(caps, dict):
+        tool_calls = cast("dict[str, Any]", caps).get("supports_tool_calls")
+        if isinstance(tool_calls, bool):
+            profile["tool_calling"] = tool_calls
+
+    return profile
+
+
 # --- Tools and structured output ------------------------------------------