langchain-loadout 0.1.0__tar.gz

langchain_loadout-0.1.0/PKG-INFO

@@ -0,0 +1,90 @@
+Metadata-Version: 2.3
+Name: langchain-loadout
+Version: 0.1.0
+Summary: Per-turn skill selection for LangChain and LangGraph agents: a judge picks the loadout, the catalog stays out of the prompt.
+Author: deyna256
+Author-email: deyna256 <literallybugcreator@gmail.com>
+Requires-Dist: langchain>=1.0
+Requires-Dist: deepagents>=0.7.15
+Requires-Dist: typesafe-sdk>=0.7.0 ; extra == 'jev'
+Requires-Python: >=3.11
+Provides-Extra: jev
+Description-Content-Type: text/markdown
+
+# langchain-loadout
+
+<p><strong>Per-turn skill selection for LangChain and deepagents agents: the model sees the few skills
+it needs, not a catalog of hundreds.</strong></p>
+
+[![CI](https://github.com/deyna256/langchain-loadout/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/deyna256/langchain-loadout/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fdeyna256%2Flangchain-loadout%2Fmain%2Fpyproject.toml)](pyproject.toml)
+[![License: MIT](https://img.shields.io/github/license/deyna256/langchain-loadout)](LICENSE)
+
+[What it gives](#what-it-gives) · [Limits](#limits) · [How it works](docs/design.md) ·
+[Contributing](CONTRIBUTING.md)
+
+---
+
+> [!NOTE]
+> Not released yet. The library works and is measured, but the public API is still moving and there is
+> no package on PyPI.
+
+An agent with hundreds of skills carries every name and description in its system prompt, on every
+model call. Loadout decides each turn which skills matter and shows the model only those.
+
+- **Each turn stands alone.** "Is a skill needed at all" is asked alongside the ranking, so a request
+  that needs no skill costs one cheap answer and nothing is loaded. No state, no checkpointer, nothing
+  to carry between turns.
+- **The logic is Loadout's, the judge is yours.** The questions are simple — "pick one", "yes or no" —
+  and go through a single port. A ready adapter ships for
+  [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev).
+- **Confidence decides what the agent sees.** High: the skill's instructions go straight into the
+  request. Medium: two or three candidates. Low: nothing, and the model can still call `find_skill`.
+  Every threshold is a setting.
+- **A failure does not break the agent.** Loadout wraps the ordinary skills middleware. If it times out
+  or errors, the agent gets the full list, exactly as it would without Loadout.
+
+## What it gives
+
+Measured on a testbed — a bank-statement assistant with a catalog of 236 skills, an agent on
+deepagents, 50 conversations of 5 turns each:
+
+| | with Loadout | full catalog in the prompt |
+|---|---|---|
+| correct answer to the user | 86% | 86% |
+| the right skill was taken | **86%** | 57% |
+| skills section of the prompt | **5,648 characters** | 89,150 characters |
+| input tokens per turn | **34,131** | 111,864 |
+
+When the agent has tools and can work the answer out for itself, the skill barely affects whether the
+answer is right. What Loadout delivers consistently is context and a predictable skill choice. The
+reasoning behind the design is in [docs/design.md](docs/design.md).
+
+## Limits
+
+- **It is not an accuracy feature.** Where a skill only restates what the model could work out, the
+  answer is the same either way.
+- **A decision costs about 3 s on a catalog of 236 skills.** Two seconds is reachable on a catalog of
+  about a hundred, or on a turn that continues a topic.
+- **A turn costs slightly more, not less.** The full catalog is identical every message and caches
+  well; the Loadout prompt changes every turn and does not.
+- **Thresholds have to be fitted on your own data**, and the library has no procedure for that yet.
+- **Everything above was measured on generated data**, with one judge and one agent model.
+
+The reasoning behind each of these is in [docs/design.md](docs/design.md#known-limits).
+
+## Development
+
+```sh
+uv sync
+just test     # the test suite; no network needed
+just lint     # formatting, style and import order
+just type     # types
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for issues, branches, commits and reviews, and
+[docs/development.md](docs/development.md) for the coding rules.
+
+## License
+
+[MIT](LICENSE)

langchain_loadout-0.1.0/README.md

@@ -0,0 +1,77 @@
+# langchain-loadout
+
+<p><strong>Per-turn skill selection for LangChain and deepagents agents: the model sees the few skills
+it needs, not a catalog of hundreds.</strong></p>
+
+[![CI](https://github.com/deyna256/langchain-loadout/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/deyna256/langchain-loadout/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fdeyna256%2Flangchain-loadout%2Fmain%2Fpyproject.toml)](pyproject.toml)
+[![License: MIT](https://img.shields.io/github/license/deyna256/langchain-loadout)](LICENSE)
+
+[What it gives](#what-it-gives) · [Limits](#limits) · [How it works](docs/design.md) ·
+[Contributing](CONTRIBUTING.md)
+
+---
+
+> [!NOTE]
+> Not released yet. The library works and is measured, but the public API is still moving and there is
+> no package on PyPI.
+
+An agent with hundreds of skills carries every name and description in its system prompt, on every
+model call. Loadout decides each turn which skills matter and shows the model only those.
+
+- **Each turn stands alone.** "Is a skill needed at all" is asked alongside the ranking, so a request
+  that needs no skill costs one cheap answer and nothing is loaded. No state, no checkpointer, nothing
+  to carry between turns.
+- **The logic is Loadout's, the judge is yours.** The questions are simple — "pick one", "yes or no" —
+  and go through a single port. A ready adapter ships for
+  [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev).
+- **Confidence decides what the agent sees.** High: the skill's instructions go straight into the
+  request. Medium: two or three candidates. Low: nothing, and the model can still call `find_skill`.
+  Every threshold is a setting.
+- **A failure does not break the agent.** Loadout wraps the ordinary skills middleware. If it times out
+  or errors, the agent gets the full list, exactly as it would without Loadout.
+
+## What it gives
+
+Measured on a testbed — a bank-statement assistant with a catalog of 236 skills, an agent on
+deepagents, 50 conversations of 5 turns each:
+
+| | with Loadout | full catalog in the prompt |
+|---|---|---|
+| correct answer to the user | 86% | 86% |
+| the right skill was taken | **86%** | 57% |
+| skills section of the prompt | **5,648 characters** | 89,150 characters |
+| input tokens per turn | **34,131** | 111,864 |
+
+When the agent has tools and can work the answer out for itself, the skill barely affects whether the
+answer is right. What Loadout delivers consistently is context and a predictable skill choice. The
+reasoning behind the design is in [docs/design.md](docs/design.md).
+
+## Limits
+
+- **It is not an accuracy feature.** Where a skill only restates what the model could work out, the
+  answer is the same either way.
+- **A decision costs about 3 s on a catalog of 236 skills.** Two seconds is reachable on a catalog of
+  about a hundred, or on a turn that continues a topic.
+- **A turn costs slightly more, not less.** The full catalog is identical every message and caches
+  well; the Loadout prompt changes every turn and does not.
+- **Thresholds have to be fitted on your own data**, and the library has no procedure for that yet.
+- **Everything above was measured on generated data**, with one judge and one agent model.
+
+The reasoning behind each of these is in [docs/design.md](docs/design.md#known-limits).
+
+## Development
+
+```sh
+uv sync
+just test     # the test suite; no network needed
+just lint     # formatting, style and import order
+just type     # types
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for issues, branches, commits and reviews, and
+[docs/development.md](docs/development.md) for the coding rules.
+
+## License
+
+[MIT](LICENSE)

langchain_loadout-0.1.0/pyproject.toml

@@ -0,0 +1,50 @@
+[project]
+name = "langchain-loadout"
+version = "0.1.0"
+description = "Per-turn skill selection for LangChain and LangGraph agents: a judge picks the loadout, the catalog stays out of the prompt."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "langchain>=1.0",
+    "deepagents>=0.7.15",
+]
+
+[[project.authors]]
+name = "deyna256"
+email = "literallybugcreator@gmail.com"
+
+[project.optional-dependencies]
+jev = ["typesafe-sdk>=0.7.0"]
+
+[build-system]
+requires = ["uv_build>=0.12.16,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "pytest-asyncio>=1.4.0",
+    "ruff>=0.16.8",
+    "ty>=0.0.82",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 135
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "W",
+    "F",
+    "I",
+    "B",
+    "C4",
+    "UP",
+    "SIM",
+    "RUF",
+]

langchain_loadout-0.1.0/pyproject.toml.orig

@@ -0,0 +1,39 @@
+[project]
+name = "langchain-loadout"
+version = "0.1.0"
+description = "Per-turn skill selection for LangChain and LangGraph agents: a judge picks the loadout, the catalog stays out of the prompt."
+readme = "README.md"
+authors = [
+    { name = "deyna256", email = "literallybugcreator@gmail.com" }
+]
+requires-python = ">=3.11"
+dependencies = [
+    "langchain>=1.0",
+    "deepagents>=0.7.15",
+]
+
+[project.optional-dependencies]
+jev = ["typesafe-sdk>=0.7.0"]
+
+[build-system]
+requires = ["uv_build>=0.12.16,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "pytest-asyncio>=1.4.0",
+    "ruff>=0.16.8",
+    "ty>=0.0.82",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 135
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "C4", "UP", "SIM", "RUF"]

langchain_loadout-0.1.0/src/langchain_loadout/__init__.py

@@ -0,0 +1,46 @@
+"""Per-turn skill selection for LangChain and deepagents agents.
+
+Everything listed in `__all__` is the public API; anything else may change without notice.
+
+The root exports the core only, so importing the package pulls in no framework and no provider SDK.
+The parts that do are imported from their own modules:
+
+- `langchain_loadout.langchain.middleware` — `LoadoutSkillsMiddleware`, which replaces
+  `SkillsMiddleware` in a deepagents agent;
+- `langchain_loadout.providers.jev` — `JevJudge`, the adapter for Jev (TypeSafe System One), which
+  needs the `jev` extra and `TYPESAFE_API_KEY`.
+
+Start from the README; the reasoning behind the design is in docs/design.md.
+"""
+
+from langchain_loadout.core.judge import (
+    Answer,
+    Judge,
+    JudgeMisconfigured,
+    JudgeUnavailable,
+    Limits,
+    LoadoutError,
+    Pick,
+    YesNo,
+)
+from langchain_loadout.core.router import SkillRouter, decide_from_trace
+from langchain_loadout.core.types import DEFAULTS, Decision, Settings, Skill, Trace, Turn
+
+__all__ = [
+    "DEFAULTS",
+    "Answer",
+    "Decision",
+    "Judge",
+    "JudgeMisconfigured",
+    "JudgeUnavailable",
+    "Limits",
+    "LoadoutError",
+    "Pick",
+    "Settings",
+    "Skill",
+    "SkillRouter",
+    "Trace",
+    "Turn",
+    "YesNo",
+    "decide_from_trace",
+]

langchain_loadout-0.1.0/src/langchain_loadout/core/__init__.py

@@ -0,0 +1,5 @@
+"""The framework-free core: the data, the judge port and the router that decides a turn.
+
+Nothing here imports langchain, deepagents or a provider SDK. Its public names are re-exported from
+`langchain_loadout`, which is where callers should import them from.
+"""

langchain_loadout-0.1.0/src/langchain_loadout/core/judge.py

@@ -0,0 +1,83 @@
+"""The judge port: simple questions in our own vocabulary, which an adapter translates for a provider."""
+
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Protocol
+
+
+@dataclass(frozen=True)
+class Pick:
+    """Pick one option. `options` maps a key to its description."""
+
+    instructions: str
+    options: Mapping[str, str]
+
+
+@dataclass(frozen=True)
+class YesNo:
+    """Yes or no."""
+
+    instructions: str
+
+
+@dataclass(frozen=True)
+class Answer:
+    """For `Pick`, option to probability. For `YesNo`, {"yes": probability of yes}."""
+
+    probabilities: Mapping[str, float]
+
+    @property
+    def yes(self) -> float:
+        return self.probabilities["yes"]
+
+
+@dataclass(frozen=True)
+class Limits:
+    """A provider's limits for one call. The defaults mean the provider declared none."""
+
+    max_tokens: int = 1_000_000  # the state plus the longest question
+    max_options: int = 1_000_000  # options in a single choice
+    tokens_per_char: float = 1.0  # upper estimate of tokens per character of text
+
+
+class LoadoutError(Exception):
+    """Base for every error this library raises."""
+
+
+class JudgeUnavailable(LoadoutError):
+    """The judge could not answer this time: a network problem, a timeout, a provider error.
+
+    The router treats this as a failed turn and falls back: the agent gets the full catalog, exactly as
+    it would without Loadout.
+    """
+
+
+class JudgeMisconfigured(LoadoutError):
+    """The judge cannot work at all: missing or rejected credentials, or an adapter that does not meet
+    the contract.
+
+    Retrying does not help, so this is never swallowed. It surfaces to the application instead of
+    turning into a fallback that hides a broken setup for the life of the process.
+    """
+
+
+class Judge(Protocol):
+    """What an adapter has to provide for the router to work.
+
+    `ask` answers every question in one call: that is where the cost saving comes from, and an adapter
+    for a provider that answers one at a time has to fan out internally. The state is a mapping of
+    JSON-serialisable values. A `Pick` is answered with a probability for every option, not just the
+    winner, because the router ranks candidates from that distribution, and the probabilities are
+    compared against thresholds, so they have to be calibrated and within [0, 1].
+
+    An adapter also declares `limits`, which the router uses to split a catalog that does not fit one
+    call. Declare `Limits()` to say the provider has none; leaving it out is an error, because a silent
+    default would split blind.
+
+    Failures are reported as `JudgeUnavailable` when a retry could help and `JudgeMisconfigured` when it
+    could not. Anything else an adapter raises is treated as unavailable.
+    """
+
+    limits: Limits
+
+    async def ask(self, state: Mapping[str, object], questions: Mapping[str, Pick | YesNo]) -> Mapping[str, Answer]: ...

langchain_loadout-0.1.0/src/langchain_loadout/core/router.py

@@ -0,0 +1,224 @@
+"""The skill router: every decision rule lives here, and the provider sits behind the `Judge` port.
+
+A turn goes through:
+
+1. Ranking, with "is a skill needed at all" asked alongside it: pick from the catalog by name and
+   description, keeping the best `max_candidates`. If the catalog does not fit the provider's limits
+   (`judge.limits`), it is split into parts that do, keeping groups together; the parts are asked in
+   parallel and their winners merged by a further pick, repeating until everything fits one call.
+   A low "need" ends the decision here, with nothing loaded.
+2. Verification: does each candidate fit, judged from the head of its text. With `skip_verify_at` set
+   and ranking confident in its first candidate, this step is skipped.
+3. Thresholds from `Settings`, applied by `decide_from_trace`.
+
+A turn is decided on its own. Nothing is carried over from the turn before it; why not is on `Turn`.
+
+The questions are written in English, which is what the judges measured so far answer best in; the
+user's own request is passed through unchanged, in whatever language it arrives. A product can replace
+the one question that depends on its domain through `Settings.need_question`.
+"""
+
+import asyncio
+import json
+import time
+from collections.abc import Mapping, Sequence
+from dataclasses import replace
+
+from langchain_loadout.core.judge import Judge, JudgeMisconfigured, Limits, Pick, YesNo
+from langchain_loadout.core.types import DEFAULTS, Decision, Settings, Skill, Trace, Turn
+
+# Every question the judge is asked lives in `Settings`, so a product can write them for its own domain.
+# A candidate's text goes in its own question rather than in the shared state. That way its score does
+# not depend on its neighbours in the same call (measured: a shift of up to 0.46 with neighbours in the
+# state, no more than 0.05 this way), and a call costs the state plus one longest question rather than
+# every candidate at once.
+FALLBACK_SUGGEST = 3  # how many ranked candidates to suggest when verification fails
+SPARE_CHARS = 1200  # room for the question texts themselves when checking whether settings fit
+
+Ranking = list[tuple[str, float]]
+
+
+def decide_from_trace(s: Settings, trace: Trace) -> Decision:
+    """Thresholds to a decision. Kept apart from the calls, so thresholds can be refitted on recorded
+    traces without asking the provider again."""
+    if trace.need is None or trace.need < s.need_at:
+        return Decision(trace=trace)
+    top = next(iter(trace.candidates), None)
+    if s.skip_verify_at is not None and top and top[1] >= s.skip_verify_at:
+        return Decision(load=(top[0],), trace=trace)  # ranking is sure; skip verification
+    by_fit = sorted(trace.fits, key=trace.fits.__getitem__, reverse=True)
+    load = tuple([n for n in by_fit if trace.fits[n] >= s.load_at][: s.max_load])
+    suggest = tuple(n for n in by_fit if n not in load and trace.fits[n] >= s.suggest_at)
+    return Decision(load=load, suggest=suggest, trace=trace)
+
+
+def call_size(state: Mapping[str, object], questions: Mapping[str, Pick | YesNo]) -> int:
+    """The size of a call in characters, erring high: measured both as JSON and as text, the larger wins."""
+    state_chars = max(len(json.dumps(state, ensure_ascii=False)), len(str(state)))
+    return state_chars + sum(
+        len(q.instructions) + sum(len(k) + len(v) + 6 for k, v in getattr(q, "options", {}).items()) for q in questions.values()
+    )
+
+
+class SkillRouter:
+    def __init__(self, catalog: Sequence[Skill], judge: Judge, settings: Settings = DEFAULTS) -> None:
+        names = [s.name for s in catalog]
+        if len(set(names)) != len(names):
+            raise ValueError(f"duplicate skill names in the catalog: {sorted({n for n in names if names.count(n) > 1})}")
+        if empty := [s.name for s in catalog if not s.description.strip()]:
+            raise ValueError(f"skills without a description: {empty}")
+        self.skills = {s.name: s for s in catalog}
+        self.judge = judge
+        self.settings = settings
+        # The protocol declares `limits`, but nothing enforces a Protocol at run time, so an adapter
+        # that forgets it is caught here rather than deep inside catalog splitting.
+        limits = getattr(judge, "limits", None)
+        if not isinstance(limits, Limits):
+            raise JudgeMisconfigured(
+                f"{type(judge).__name__} does not declare limits; add `limits = Limits(...)`, or `Limits()` if the provider has none"
+            )
+        self.limits: Limits = limits
+        self.budget = self.limits.max_tokens * settings.budget_share / self.limits.tokens_per_char  # characters per call
+        check = settings.request_chars + settings.context_chars + settings.head_chars + SPARE_CHARS
+        if check > self.budget:
+            raise JudgeMisconfigured(
+                f"settings do not fit the provider's limit: verification needs about {check} characters, "
+                f"the limit is about {int(self.budget)}; reduce head_chars, context_chars or request_chars"
+            )
+
+    async def decide(self, turn: Turn) -> Decision:
+        """Decide one turn. Any failure means changing nothing, or suggesting the ranked candidates if it was
+        verification that failed."""
+        started = time.monotonic()
+        # `_decide` appends to this as soon as ranking finishes. The timeout below fires outside `_decide`,
+        # so this is how the handler learns what had already been ranked when the clock ran out.
+        ranked_so_far: list[str] = []
+        try:
+            async with asyncio.timeout(self.settings.timeout):
+                return await self._decide(turn, started, ranked_so_far)
+        except JudgeMisconfigured:
+            raise  # retrying will not help, and a hidden broken setup is worse than a loud one
+        except Exception as err:
+            failure = "timeout" if isinstance(err, TimeoutError) else f"{type(err).__name__}: {err}"
+            trace = Trace(failure=failure, seconds=time.monotonic() - started)
+            return Decision(suggest=tuple(ranked_so_far[:FALLBACK_SUGGEST]), trace=trace)
+
+    async def _decide(self, turn: Turn, started: float, ranked_so_far: list[str]) -> Decision:
+        s = self.settings
+        base = {"request": turn.request[: s.request_chars], "context": turn.context[-s.context_chars :]}
+
+        def elapsed() -> float:
+            return time.monotonic() - started
+
+        # "Is a skill needed at all" goes out alongside the ranking rather than before it. Asked first it
+        # would end about one turn in fifty on its own and cost every other turn a round trip; asked in
+        # parallel it costs neither.
+        gate, ranked = await asyncio.gather(
+            self.judge.ask(base, {"need": YesNo(s.need_question)}),
+            self._rank(base, list(self.skills), s.max_candidates),
+        )
+        ranking, parts = ranked
+        trace = Trace(need=gate["need"].yes, parts=parts, stage="gate")
+        if trace.need is not None and trace.need < s.need_at:
+            return decide_from_trace(s, replace(trace, seconds=elapsed()))
+
+        candidates = ranking[: s.max_candidates]
+        ranked_so_far += [n for n, _ in candidates]
+        trace = replace(trace, candidates=tuple(candidates))
+
+        # Verify candidates against their texts, unless the product allows skipping a confident ranking.
+        if s.skip_verify_at is not None and candidates and candidates[0][1] >= s.skip_verify_at:
+            return decide_from_trace(s, replace(trace, stage="skip", seconds=elapsed()))
+        heads = await self._read_heads(ranked_so_far)
+        questions = {f"fits:{n}": YesNo(s.fits_question.format(name=n, text=head)) for n, head in heads.items()}
+        answers = await self.judge.ask(base, questions) if questions else {}
+        fits = {n: answers[f"fits:{n}"].yes for n in heads}
+        return decide_from_trace(s, replace(trace, fits=fits, stage="verify", seconds=elapsed()))
+
+    async def search(self, query: str, limit: int = 5) -> list[Skill]:
+        """Back the `find_skill` tool: the best skills for the model's own query, or nothing on failure."""
+        try:
+            async with asyncio.timeout(self.settings.timeout):
+                ranking, _ = await self._rank({"request": query[: self.settings.request_chars]}, list(self.skills), limit)
+        except JudgeMisconfigured:
+            raise
+        except Exception:
+            return []
+        return [self.skills[n] for n, _ in ranking][:limit]
+
+    # --- ranking that fits the provider's limits ----------------------------------------------------
+
+    async def _rank(self, state: Mapping[str, object], names: list[str], k: int) -> tuple[Ranking, int]:
+        """The best skills among `names`, best first. Too big for one call means parts, then a merge."""
+        parts = self._split_catalog(state, names)
+        answers = await asyncio.gather(*(self._ask_part(state, p) for p in parts))
+        rankings = [r for got, _ in answers for r in got]
+        if not rankings:
+            raise next(err for _, err in answers if err is not None)  # the provider's own error belongs in the trace
+        if len(rankings) == 1:
+            return rankings[0], len(parts)
+        # take the best k from each part, but fewer than the part holds, so the merge round is smaller
+        best = [n for ranking in rankings for n, _ in ranking[: max(1, min(k, len(ranking) - 1))]]
+        if len(best) >= len(names):
+            raise JudgeMisconfigured(
+                "the catalog cannot be split to fit the provider's limits: a single skill's name and "
+                "description are larger than one call allows"
+            )
+        merged, _ = await self._rank(state, best, k)
+        return merged, len(parts)
+
+    async def _ask_part(self, state: Mapping[str, object], names: list[str]) -> tuple[list[Ranking], Exception | None]:
+        """Pick within one part. A refusal is retried as two halves, which covers an underestimated size.
+        Returns the rankings obtained, and the original error when none were."""
+        try:
+            return [await self._pick(state, names)], None
+        except JudgeMisconfigured:
+            raise
+        except Exception as err:
+            if len(names) < 2:
+                return [], err
+            halves = [names[: len(names) // 2], names[len(names) // 2 :]]
+            results = await asyncio.gather(*(self._pick(state, h) for h in halves), return_exceptions=True)
+            ok = [r for r in results if not isinstance(r, BaseException)]
+            return ok, None if ok else err
+
+    async def _pick(self, state: Mapping[str, object], names: list[str]) -> Ranking:
+        options = {n: self.skills[n].description for n in names}
+        picked = await self.judge.ask(state, {"skill": Pick(self.settings.rank_question, options)})
+        ranked = sorted(picked["skill"].probabilities.items(), key=lambda kv: kv[1], reverse=True)
+        return [(n, p) for n, p in ranked if n in options]
+
+    def _split_catalog(self, state: Mapping[str, object], names: list[str]) -> list[list[str]]:
+        """Parts that each fit the limit, keeping skills of one group together where possible."""
+        fixed = call_size(state, {"skill": Pick(self.settings.rank_question, {})})
+        room = self.budget - fixed
+        cost = {n: len(n) + len(self.skills[n].description) + 6 for n in names}
+        if sum(cost.values()) <= room and len(names) <= self.limits.max_options:
+            return [names]
+        groups: dict[str, list[str]] = {}
+        for n in names:
+            groups.setdefault(self.skills[n].group or f"_{n}", []).append(n)
+        parts: list[list[str]] = [[]]
+        for members in groups.values():
+            for part in self._split_group(members, cost, room):
+                if parts[-1] and not self._fits_one_call(parts[-1] + part, cost, room):
+                    parts.append([])
+                parts[-1] += part
+        return [p for p in parts if p]
+
+    def _fits_one_call(self, names: list[str], cost: Mapping[str, int], room: float) -> bool:
+        return sum(cost[n] for n in names) <= room and len(names) <= self.limits.max_options
+
+    def _split_group(self, names: list[str], cost: Mapping[str, int], room: float) -> list[list[str]]:
+        """Split `names` into consecutive chunks that each fit `room`."""
+        parts: list[list[str]] = [[]]
+        for n in names:
+            if parts[-1] and not self._fits_one_call(parts[-1] + [n], cost, room):
+                parts.append([])
+            parts[-1].append(n)
+        return parts
+
+    async def _read_heads(self, names: list[str]) -> dict[str, str]:
+        """The head of each candidate's text. A skill that cannot be read drops out; the rest are verified."""
+        texts = await asyncio.gather(*(self.skills[n].read() for n in names), return_exceptions=True)
+        return {n: t[: self.settings.head_chars] for n, t in zip(names, texts, strict=True) if isinstance(t, str)}

langchain_loadout-0.1.0/src/langchain_loadout/core/types.py

@@ -0,0 +1,82 @@
+"""The data the router exchanges with the product. Knows nothing about providers or frameworks."""
+
+from collections.abc import Awaitable, Callable, Mapping
+from dataclasses import dataclass, field
+
+
+@dataclass(frozen=True)
+class Skill:
+    """A skill from the product's catalog. Loadout does not discover skills; it is handed them."""
+
+    name: str
+    description: str
+    read: Callable[[], Awaitable[str]]  # the full instructions; called for candidates only
+    group: str | None = None
+
+
+@dataclass(frozen=True)
+class Turn:
+    """One turn of a conversation: everything a decision needs. The product assembles the context.
+
+    A turn carries no memory of the ones before it. Whether a skill was loaded earlier cannot be known
+    reliably — graph state does not survive an invocation without a checkpointer, and a `read_file` call
+    stays visible in the history after middleware has clipped the text it fetched. Getting that wrong in
+    the "already loaded" direction would leave the agent answering without the procedure, silently.
+    """
+
+    request: str
+    context: str = ""
+
+
+@dataclass(frozen=True)
+class Settings:
+    """Everything a product may want to tune."""
+
+    max_candidates: int = 6  # how many candidates ranking passes to verification
+    max_load: int = 2  # how many skills may be loaded in one turn
+    load_at: float = 0.8  # "fits" at or above this: put the instructions in the request
+    suggest_at: float = 0.4  # "fits" at or above this: offer the skill as a candidate
+    need_at: float = 0.3  # "a skill is needed at all" below this: load nothing
+    skip_verify_at: float | None = None  # ranking this sure of its first candidate: load without verifying
+    head_chars: int = 1500  # how much of a skill's text verification sees
+    timeout: float = 2.0  # seconds for the whole decision
+    request_chars: int = 2000  # how much of the request the judge sees (a user may paste a whole statement)
+    context_chars: int = 4000  # how much of the context the judge sees, counted from the end
+    budget_share: float = 0.85  # share of the provider's limit to use, leaving room for estimation error
+    # Both questions the judge is asked. The defaults are written for a general assistant; wording that
+    # names the product's own domain separates better (measured on bank requests, the need question went
+    # from 0.80-0.97 against 0.01-0.02). The judge sees `request` and `context` in the state it is given.
+    need_question: str = (
+        "Does answering the user's request need a specialised skill (a documented procedure or work on the "
+        "user's accounts, statements or documents) rather than a plain conversational reply?"
+    )
+    rank_question: str = "Which skill's instructions would help the assistant answer the user's request best?"
+    # `fits_question` is formatted with the candidate's `name` and the head of its `text`.
+    fits_question: str = (
+        'Do these skill instructions do what the user asks for in `request`?\n\n<skill name="{name}">\n{text}\n</skill>'
+    )
+
+
+@dataclass(frozen=True)
+class Trace:
+    """What was asked, what came back and what broke. Recorded for measurement and debugging."""
+
+    candidates: tuple[tuple[str, float], ...] = ()  # the best of the ranking, with probabilities, best first
+    need: float | None = None
+    fits: Mapping[str, float] = field(default_factory=dict)  # per candidate: does it do what the request asks
+    failure: str | None = None
+    seconds: float = 0.0
+    parts: int = 1  # how many parts the catalog was split into for ranking
+    stage: str = ""  # where the decision ended: gate (no skill needed), skip (no verification) or verify
+
+
+@dataclass(frozen=True)
+class Decision:
+    """What to do on this turn."""
+
+    load: tuple[str, ...] = ()  # instructions go straight into the request
+    suggest: tuple[str, ...] = ()  # listed as candidates for the model to choose from
+    trace: Trace = field(default_factory=Trace)
+
+
+DEFAULTS = Settings()  # Settings is frozen, so one shared instance is enough

langchain_loadout-0.1.0/src/langchain_loadout/langchain/__init__.py

@@ -0,0 +1,5 @@
+"""Integration with LangChain agents. Importing this package pulls in langchain and deepagents."""
+
+from langchain_loadout.langchain.middleware import LoadoutSkillsMiddleware, recent_context
+
+__all__ = ["LoadoutSkillsMiddleware", "recent_context"]

langchain_loadout-0.1.0/src/langchain_loadout/langchain/middleware.py

@@ -0,0 +1,164 @@
+"""Loadout for deepagents: a wrapper around the ordinary skills middleware (`SkillsMiddleware`).
+
+Skills are still discovered by the ordinary middleware, through `state["skills_metadata"]`. On each new
+user message Loadout decides which of them are needed, and the model sees only those; the text of the
+loaded ones goes straight into the system message. On any failure the request passes to the ordinary
+middleware untouched, so the model sees the full list exactly as it would without Loadout.
+
+Wiring: `create_deep_agent(..., skills=[...], middleware=[LoadoutSkillsMiddleware(...)])`. The wrapper
+carries the built-in middleware's name and takes its place. The optimisation applies when the agent is
+run asynchronously (`ainvoke`, `astream`); a synchronous run takes the ordinary path.
+"""
+
+from collections.abc import Awaitable, Callable, Sequence
+from typing import Annotated, Any, NotRequired, cast
+
+from deepagents.backends.protocol import BackendProtocol
+from deepagents.middleware._utils import append_to_system_message
+from deepagents.middleware.skills import SkillMetadata, SkillsMiddleware, SkillsState
+from langchain.agents.middleware.types import ModelRequest, ModelResponse, PrivateStateAttr
+from langchain_core.messages import AnyMessage, HumanMessage
+from langchain_core.tools import BaseTool, StructuredTool
+
+from langchain_loadout.core.judge import Judge
+from langchain_loadout.core.router import SkillRouter
+from langchain_loadout.core.types import DEFAULTS, Decision, Settings, Skill, Turn
+
+PROMPT = """## Skills System
+
+Skills picked for the current request (only these are listed; the full catalog is larger):
+
+{skills_list}
+
+{loaded}If none of the listed skills fits the task, call `find_skill` with a short description of what you need.
+To use a listed skill that is not loaded below, read its SKILL.md with `read_file` (pass `limit=1000`)."""
+
+LOADED = "**Loaded skill instructions — follow them:**\n\n{texts}\n\n"
+
+
+class LoadoutState(SkillsState):
+    loadout_turn: NotRequired[Annotated[str, PrivateStateAttr]]  # id of the user message this decision was made for
+    # These three carry the decision from `abefore_model` to `awrap_model_call` within one invocation of
+    # the graph. Nothing is read back from a previous turn, so no checkpointer is required.
+    loadout_failed: NotRequired[Annotated[bool, PrivateStateAttr]]  # a failure means the ordinary full list
+    loadout_loaded: NotRequired[Annotated[list[str], PrivateStateAttr]]  # the text of these skills goes into the request
+    loadout_suggest: NotRequired[Annotated[list[str], PrivateStateAttr]]  # these are listed only
+
+
+def recent_context(messages: Sequence[AnyMessage], limit: int = 6) -> str:
+    """The default context: the messages leading up to the current user message."""
+    lines = [f"{m.type}: {str(m.content)[:500]}" for m in messages[-limit:] if str(m.content).strip()]
+    return "\n".join(lines)
+
+
+class LoadoutSkillsMiddleware(SkillsMiddleware):
+    state_schema = LoadoutState
+
+    def __init__(
+        self,
+        *,
+        backend: BackendProtocol,
+        sources: Sequence[str],
+        judge: Judge,
+        settings: Settings = DEFAULTS,
+        catalog_hint: str = "",
+        context: Callable[[Sequence[AnyMessage]], str] = recent_context,
+        on_decision: Callable[[Decision], None] | None = None,
+    ) -> None:
+        super().__init__(backend=backend, sources=sources)
+        self.judge, self.settings, self.context = judge, settings, context
+        self.on_decision = on_decision  # every decision's trace, for logs, metrics and measurement
+        self._router: SkillRouter | None = None
+        self._router_for: tuple[str, ...] = ()
+        self._paths: dict[str, str] = {}
+        self.tools: list[BaseTool] = [self._find_skill_tool(catalog_hint)]
+
+    @property
+    def name(self) -> str:
+        return "SkillsMiddleware"  # takes the place of the built-in deepagents middleware
+
+    # --- the decision: once per new user message ----------------------------------------------------
+
+    async def abefore_model(self, state: SkillsState, runtime: Any) -> dict[str, Any] | None:
+        # The base class fixes this parameter to SkillsState, while `state_schema = LoadoutState` is what
+        # the graph actually builds, so the narrowing has to be stated here rather than in the signature.
+        ours = cast(LoadoutState, state)
+        messages = ours["messages"]
+        last = next((i for i in range(len(messages) - 1, -1, -1) if isinstance(messages[i], HumanMessage)), None)
+        if last is None:
+            return None
+        turn_id = messages[last].id or str(last)
+        if ours.get("loadout_turn") == turn_id:
+            return None  # this turn already has a decision
+        router = self._router_from(ours.get("skills_metadata", []))
+        turn = Turn(request=str(messages[last].content), context=self.context(messages[:last]))
+        decision = await router.decide(turn)
+        if self.on_decision:
+            self.on_decision(decision)
+        if decision.trace.failure:
+            return {"loadout_turn": turn_id, "loadout_failed": True}
+        return {
+            "loadout_turn": turn_id,
+            "loadout_failed": False,
+            "loadout_loaded": list(decision.load),
+            "loadout_suggest": list(decision.suggest),
+        }
+
+    # --- applying it: what the model sees -----------------------------------------------------------
+
+    async def awrap_model_call(
+        self, request: ModelRequest, handler: Callable[[ModelRequest], Awaitable[ModelResponse]]
+    ) -> ModelResponse:
+        state = request.state
+        if "loadout_turn" not in state or state.get("loadout_failed"):
+            return await super().awrap_model_call(request, handler)  # the ordinary path: the full list
+        loaded = state.get("loadout_loaded", [])
+        picked = set(loaded) | set(state.get("loadout_suggest", []))
+        listed: list[SkillMetadata] = [m for m in state.get("skills_metadata", []) if m["name"] in picked]
+        texts = [await self._text(name) for name in loaded if name in self._paths]
+        section = PROMPT.format(
+            skills_list=self._format_skills_list(listed) if listed else "(nothing picked for this request)",
+            loaded=LOADED.format(texts="\n\n---\n\n".join(texts)) if texts else "",
+        )
+        return await handler(request.override(system_message=append_to_system_message(request.system_message, section)))
+
+    # --- catalog and search -------------------------------------------------------------------------
+
+    def _router_from(self, metadata: list[SkillMetadata]) -> SkillRouter:
+        names = tuple(m["name"] for m in metadata)
+        if self._router is None or names != self._router_for:
+            self._paths = {m["name"]: m["path"] for m in metadata}
+            catalog = [Skill(m["name"], m["description"], self._reader(m["name"])) for m in metadata]
+            self._router, self._router_for = SkillRouter(catalog, self.judge, self.settings), names
+        return self._router
+
+    def _reader(self, name: str) -> Callable[[], Awaitable[str]]:
+        async def read() -> str:
+            return await self._text(name)
+
+        return read
+
+    async def _text(self, name: str) -> str:
+        """Read a skill's instructions from the backend. Deliberately not cached: an agent that runs for
+        days would otherwise keep serving the text a SKILL.md had when it first read it."""
+        [response] = await self._backend.adownload_files([self._paths[name]])
+        if response.error or response.content is None:
+            raise OSError(f"could not read {self._paths[name]}: {response.error}")
+        return response.content.decode()
+
+    def _find_skill_tool(self, catalog_hint: str) -> BaseTool:
+        async def find_skill(query: str) -> str:
+            if self._router is None:
+                return "Skill catalog is not loaded yet."
+            found = await self._router.search(query)
+            if not found:
+                return "No matching skill found."
+            return "\n".join(
+                f"- **{s.name}**: {s.description}\n  -> Read `{self._paths[s.name]}` for full instructions" for s in found
+            )
+
+        description = (f"{catalog_hint} " if catalog_hint else "") + (
+            "Find skills (step-by-step instructions) for a task that the skills listed in the system prompt don't cover. "
+            "Pass a short description of the task; returns the best matching skills with paths to read."
+        )
+        return StructuredTool.from_function(coroutine=find_skill, name="find_skill", description=description)

langchain_loadout-0.1.0/src/langchain_loadout/providers/__init__.py

@@ -0,0 +1,5 @@
+"""Adapters that satisfy the `Judge` port for a specific provider.
+
+Each adapter lives in its own module and brings its own optional dependency, so importing this
+package costs nothing until a specific adapter is imported.
+"""

langchain_loadout-0.1.0/src/langchain_loadout/providers/jev.py

@@ -0,0 +1,61 @@
+"""Adapter for Jev (TypeSafe): turns our questions into its choices and nouls, all in one call."""
+
+from collections.abc import Callable, Mapping
+from typing import cast
+
+from typesafe_sdk import (
+    AsyncTypeSafeClient,
+    Choice,
+    ChoiceAnswer,
+    JSONContent,
+    Noul,
+    NoulAnswer,
+    RetryPolicy,
+    TypeSafeAuthenticationError,
+    TypeSafePermissionDeniedError,
+)
+
+from langchain_loadout.core.judge import Answer, JudgeMisconfigured, JudgeUnavailable, Limits, Pick, YesNo
+
+# Credentials that the provider rejects: retrying cannot help, so these surface instead of falling back.
+# A bad request, including one that is too large, is deliberately not here: the router answers that by
+# splitting the catalog and asking again.
+PERMANENT = (TypeSafeAuthenticationError, TypeSafePermissionDeniedError)
+
+
+class JevJudge:
+    # 32k tokens for the state plus the longest question, up to 255 options (docs.typesafe.ai/models).
+    # Tokens per character: Russian descriptions measure about 0.68, so 0.8 leaves a margin.
+    limits = Limits(max_tokens=32_000, max_options=255, tokens_per_char=0.8)
+
+    def __init__(self, client: AsyncTypeSafeClient | None = None, on_usage: Callable[[int], None] | None = None) -> None:
+        # No retries: a decision has two seconds, and a late answer is useless because the fallback has run.
+        self.client = client or AsyncTypeSafeClient(retry=RetryPolicy(max_retries=0))
+        self.on_usage = on_usage  # tokens per call, so cost can be attributed to decisions under concurrency
+
+    async def ask(self, state: Mapping[str, object], questions: Mapping[str, Pick | YesNo]) -> Mapping[str, Answer]:
+        asked = {key: _to_jev(q) for key, q in questions.items()}
+        try:
+            # The port requires the state to be JSON-serialisable, which is what JSONContent means.
+            response = await self.client.system_one(cast(JSONContent, dict(state)), asked)
+        except PERMANENT as err:
+            raise JudgeMisconfigured(f"Jev rejected the credentials: {err}") from err
+        except Exception as err:
+            raise JudgeUnavailable(f"Jev did not answer: {err}") from err
+        if self.on_usage:
+            self.on_usage(response.usage.input_tokens or 0)
+        return {key: _from_jev(answer) for key, answer in response.answers.items()}
+
+
+def _to_jev(q: Pick | YesNo) -> Choice | Noul:
+    if isinstance(q, Pick):
+        return Choice(instructions=q.instructions, criteria=dict(q.options))
+    return Noul(instructions=q.instructions)
+
+
+def _from_jev(answer: object) -> Answer:
+    if isinstance(answer, ChoiceAnswer):
+        return Answer(dict(answer.probabilities))
+    if isinstance(answer, NoulAnswer):
+        return Answer({"yes": answer.noul})
+    raise TypeError(f"unexpected answer from Jev: {type(answer).__name__}")

langchain_loadout-0.1.0/src/langchain_loadout/testing/__init__.py

@@ -0,0 +1,10 @@
+"""Tools for people writing a judge adapter or testing an agent that uses Loadout.
+
+`check_judge` runs an adapter against the `Judge` contract, and `ScriptedJudge` answers whatever a test
+tells it to, so an agent can be exercised without calling a provider at all.
+"""
+
+from langchain_loadout.testing.conformance import check_judge
+from langchain_loadout.testing.fakes import ScriptedJudge, yes
+
+__all__ = ["ScriptedJudge", "check_judge", "yes"]

langchain_loadout-0.1.0/src/langchain_loadout/testing/conformance.py

@@ -0,0 +1,78 @@
+"""Check a judge adapter against the contract the router relies on.
+
+An adapter author runs this against their provider:
+
+    from langchain_loadout.testing import check_judge
+
+    async def test_my_adapter():
+        await check_judge(MyJudge())
+
+It makes one real call, so it costs whatever the provider charges for one call.
+"""
+
+from typing import NoReturn
+
+from langchain_loadout.core.judge import Judge, Limits, Pick, YesNo
+
+STATE = {"request": "Show me what I spent last month, broken down by category."}
+
+SKILLS = {
+    "spending-by-category": "Break the user's spending down by category over a period.",
+    "card-limits": "Explain the limits set on the user's card.",
+    "visa-statement": "Produce a bank statement formatted for a visa application.",
+}
+
+QUESTIONS: dict[str, Pick | YesNo] = {
+    "pick": Pick("Which skill's instructions would help answer `request` best?", SKILLS),
+    "about-spending": YesNo("Is `request` about money the user has already spent?"),
+    "about-transfer": YesNo("Is `request` asking to transfer money to someone?"),
+}
+
+
+def _fail(what: str) -> NoReturn:
+    raise AssertionError(f"judge does not meet the contract: {what}")
+
+
+def _require(ok: bool, what: str) -> None:
+    if not ok:
+        _fail(what)
+
+
+def _probability(value: object, what: str) -> float:
+    if not isinstance(value, int | float):
+        _fail(f"{what} is {type(value).__name__}, expected a number")
+    number = float(value)
+    _require(0.0 <= number <= 1.0, f"{what} is {number}, expected a probability within [0, 1]")
+    return number
+
+
+async def check_judge(judge: Judge) -> None:
+    """Raise `AssertionError` unless the adapter meets every requirement of the `Judge` contract."""
+    limits = getattr(judge, "limits", None)
+    if not isinstance(limits, Limits):
+        _fail("`limits` is not declared; use `Limits()` if the provider has none")
+    _require(limits.max_tokens > 0, f"limits.max_tokens is {limits.max_tokens}, expected a positive number")
+    _require(limits.max_options > 1, f"limits.max_options is {limits.max_options}, expected more than one option")
+    _require(limits.tokens_per_char > 0, f"limits.tokens_per_char is {limits.tokens_per_char}, expected a positive number")
+
+    answers = await judge.ask(STATE, QUESTIONS)
+
+    _require(set(answers) == set(QUESTIONS), f"one call answered {sorted(answers)}, expected {sorted(QUESTIONS)}")
+
+    picked = answers["pick"].probabilities
+    _require(
+        set(picked) == set(SKILLS),
+        f"a Pick answered about {sorted(picked)}, expected a probability for every option: {sorted(SKILLS)}",
+    )
+    for name, value in picked.items():
+        _probability(value, f"the probability of option {name!r}")
+    best = max(picked, key=picked.__getitem__)
+    _require(best == "spending-by-category", f"the best option was {best!r}, expected 'spending-by-category'")
+
+    spending = _probability(answers["about-spending"].probabilities.get("yes"), "the answer to a YesNo question")
+    transfer = _probability(answers["about-transfer"].probabilities.get("yes"), "the answer to a YesNo question")
+    _require(
+        spending > transfer,
+        f"an obvious yes scored {spending} and an obvious no scored {transfer}; "
+        "probabilities have to be calibrated, because the router compares them against thresholds",
+    )

langchain_loadout-0.1.0/src/langchain_loadout/testing/fakes.py

@@ -0,0 +1,26 @@
+"""A judge that answers from a script instead of calling a provider."""
+
+from collections.abc import Callable, Mapping
+
+from langchain_loadout.core.judge import Answer, Limits, Pick, YesNo
+
+Questions = Mapping[str, Pick | YesNo]
+Script = Callable[[Mapping[str, object], Questions], Mapping[str, Answer]]
+
+
+class ScriptedJudge:
+    """Answers whatever `script` returns and remembers every call, so a test can assert on them."""
+
+    def __init__(self, script: Script, limits: Limits | None = None) -> None:
+        self.script = script
+        self.limits = limits or Limits()
+        self.calls: list[tuple[Mapping[str, object], Questions]] = []
+
+    async def ask(self, state: Mapping[str, object], questions: Questions) -> Mapping[str, Answer]:
+        self.calls.append((state, questions))
+        return self.script(state, questions)
+
+
+def yes(p: float) -> Answer:
+    """The answer to a `YesNo` question, with `p` as the probability of yes."""
+    return Answer({"yes": p})