evenkeel 0.1.0__tar.gz

evenkeel-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.ruff_cache/
+dist/
+evals/results/
+.DS_Store
+
+# Working documents: specs, plans and notes stay on this machine
+docs/
+.superpowers/

evenkeel-0.1.0/CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-10-13
+
+First release.
+
+### Added
+- `EvenKeelMiddleware` for LangChain agents (`create_agent`): classifies failed model and tool calls and
+  backs off, retries with a hint, fails, or escalates to a person with `interrupt()`.
+- Dollar budgets from token usage and a user-supplied `PriceTable`, with a pause for approval when over budget.
+- Attempt records in agent state (`evenkeel_attempts`).
+- `RuleClassifier` and `JevClassifier` (optional `jev` extra), ported from keel.
+- Contract tests for the LangChain, LangGraph and typesafe-sdk behaviour the middleware relies on.
+
+[0.1.0]: https://github.com/yodablocks/evenkeel/releases/tag/v0.1.0

evenkeel-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 yodablocks
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evenkeel-0.1.0/PKG-INFO

@@ -0,0 +1,144 @@
+Metadata-Version: 2.5
+Name: evenkeel
+Version: 0.1.0
+Summary: Failure classification, retries with hints, dollar budgets and escalation for LangChain agents. Ported from keel.
+Project-URL: Repository, https://github.com/yodablocks/evenkeel
+Project-URL: keel, https://github.com/yodablocks/keel
+Author: yodablocks
+License: MIT
+License-File: LICENSE
+Keywords: agents,budgets,human-in-the-loop,jev,langchain,langgraph,middleware,retries
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Requires-Dist: langchain<2,>=1.3
+Requires-Dist: langgraph<2,>=1
+Requires-Dist: typing-extensions>=4.7
+Provides-Extra: jev
+Requires-Dist: typesafe-sdk<0.8,>=0.7; extra == 'jev'
+Description-Content-Type: text/markdown
+
+# evenkeel
+
+Keeps a LangChain agent on an even keel when things fail: failures are classified and handled by kind,
+bad output is retried with a hint, spend is tracked in dollars with a budget, and failures that need a person
+pause the run for approval.
+
+Ported from [keel](https://github.com/yodablocks/keel), a durable execution engine for AI agents.
+An independent project, not affiliated with LangChain. Experimental: 0.x, expect changes.
+
+## What it adds to LangChain's built-in middleware
+
+| Need | Built in | EvenKeelMiddleware |
+|---|---|---|
+| Retries | `ModelRetryMiddleware`, `ToolRetryMiddleware` retry with the same input | Classifies the failure first: back off, retry **with a hint**, fail at once, or escalate |
+| Limits | `ModelCallLimitMiddleware` counts calls | Tracks **dollars** from token usage and pauses at a budget |
+| Approvals | `HumanInTheLoopMiddleware` asks before chosen tools | Escalates **when a failure needs a person**, from any model or tool call |
+
+Use it alongside the built-ins: `ModelFallbackMiddleware` for switching models, `HumanInTheLoopMiddleware`
+for approvals fixed in advance.
+
+## Install
+
+    pip install evenkeel          # rule-based classification
+    pip install "evenkeel[jev]"   # adds TypeSafe's Jev classifier
+
+## Use
+
+```python
+from langchain.agents import create_agent
+from langgraph.checkpoint.memory import InMemorySaver
+from evenkeel import EvenKeelMiddleware, ModelPrice, PriceTable
+
+agent = create_agent(
+    "openai:gpt-4o-mini",
+    tools=[...],
+    middleware=[
+        EvenKeelMiddleware(
+            budget_usd=0.50,
+            prices=PriceTable({"gpt-4o-mini": ModelPrice(input_per_mtok=0.15, output_per_mtok=0.60)}),
+        )
+    ],
+    checkpointer=InMemorySaver(),  # needed for pauses
+)
+```
+
+With Jev (needs `TYPESAFE_API_KEY`):
+
+```python
+from typesafe_sdk import TypeSafeClient
+from evenkeel import EvenKeelMiddleware, JevClassifier
+
+EvenKeelMiddleware(classifier=JevClassifier(TypeSafeClient()))
+```
+
+## How failures are handled
+
+| Kind | Example | Action |
+|---|---|---|
+| `transient` | 429, 503, timeout, dropped connection | Back off and retry the same call |
+| `bad_output` | Malformed output, a call that breaks the schema (recognised by Jev, or raised as `BadOutputError`) | Retry with a hint saying what was wrong |
+| `bad_input` | Input a tool rejects (`BadInputError`, HTTP 400 or 422, a pydantic `ValidationError`) | Model call: fail. Tool call: return the error to the model so it fixes its arguments |
+| `needs_human` | Refund over a limit, a policy block | Pause with `interrupt()` and wait for a decision |
+| `fatal` | A bug, bad credentials | Re-raise the original error at once |
+
+Retries and escalations stop at `max_attempts` (default 4). Each failure is recorded (error, kind, confidence,
+which classifier decided, action): in the agent's state under `evenkeel_attempts` when the call ends in a
+result, and on the raised error as `error.evenkeel_attempts` when it finally fails.
+
+Some failures never reach the middleware, because LangChain turns them into messages for the model itself:
+a call to a tool that does not exist, and tool arguments that fail the tool's schema.
+
+## Pauses and how to resume
+
+A failure that needs a person:
+
+```python
+from langgraph.types import Command
+
+agent.invoke(Command(resume={"decision": "retry", "hint": "approved by finance"}), config)
+agent.invoke(Command(resume={"decision": "fail"}), config)
+agent.invoke(Command(resume={"decision": "skip", "message": "Refunded by hand."}), config)  # tool calls only
+```
+
+A `retry` with a `hint` on a tool call sends the reviewer's words to the model, which can call the tool again; a
+tool cannot take a hint itself. A call that still needs a person after one approved retry fails. An answer
+that is not one of these values does not raise: the run pauses again, with an `error` field in the payload
+saying what was wrong.
+
+Over budget:
+
+```python
+agent.invoke(Command(resume={"approve": True, "budget_usd": 1.00}), config)
+agent.invoke(Command(resume={"approve": False}), config)  # ends the run
+```
+
+The budget applies to each run: every new `invoke` on a thread starts with zero spend and the configured
+budget.
+
+## Limitations
+
+- **Budgets can overshoot by one call.** The check runs before a model call, but a call's cost is known only
+  after it. keel has the same limitation.
+- **No built-in prices.** You supply them; a model without a price, or a response without token usage, is an
+  error rather than a silent $0. Dated or versioned names (`gpt-4o-mini-2024-07-18`, `claude-3-5-sonnet-20241022`, `gpt-4-0613`, `...-latest`)
+  use the price of their base name; any other variant (`gpt-4o-mini` against `gpt-4`, `gpt-4-turbo`) needs its own
+  entry.
+- **On resume after an escalation, the failing call runs once more** before the decision applies, because
+  LangGraph re-runs the paused node. For a tool with side effects, make it idempotent or use `skip`.
+- **Classification is only as good as the classifier.** Rules only recognise explicit signals (status codes,
+  network errors, evenkeel's error classes): on keel's 75 error messages from public SDK issues they pick the
+  right kind 29 times and the right action 39 times. Jev reads the message itself; see
+  [keel's results](https://github.com/yodablocks/keel#failure-classification-rules-vs-jev) for its accuracy on
+  real error messages, and `evals/` to measure it here.
+
+## Develop
+
+    uv sync --all-extras
+    uv run pytest

evenkeel-0.1.0/README.md

@@ -0,0 +1,118 @@
+# evenkeel
+
+Keeps a LangChain agent on an even keel when things fail: failures are classified and handled by kind,
+bad output is retried with a hint, spend is tracked in dollars with a budget, and failures that need a person
+pause the run for approval.
+
+Ported from [keel](https://github.com/yodablocks/keel), a durable execution engine for AI agents.
+An independent project, not affiliated with LangChain. Experimental: 0.x, expect changes.
+
+## What it adds to LangChain's built-in middleware
+
+| Need | Built in | EvenKeelMiddleware |
+|---|---|---|
+| Retries | `ModelRetryMiddleware`, `ToolRetryMiddleware` retry with the same input | Classifies the failure first: back off, retry **with a hint**, fail at once, or escalate |
+| Limits | `ModelCallLimitMiddleware` counts calls | Tracks **dollars** from token usage and pauses at a budget |
+| Approvals | `HumanInTheLoopMiddleware` asks before chosen tools | Escalates **when a failure needs a person**, from any model or tool call |
+
+Use it alongside the built-ins: `ModelFallbackMiddleware` for switching models, `HumanInTheLoopMiddleware`
+for approvals fixed in advance.
+
+## Install
+
+    pip install evenkeel          # rule-based classification
+    pip install "evenkeel[jev]"   # adds TypeSafe's Jev classifier
+
+## Use
+
+```python
+from langchain.agents import create_agent
+from langgraph.checkpoint.memory import InMemorySaver
+from evenkeel import EvenKeelMiddleware, ModelPrice, PriceTable
+
+agent = create_agent(
+    "openai:gpt-4o-mini",
+    tools=[...],
+    middleware=[
+        EvenKeelMiddleware(
+            budget_usd=0.50,
+            prices=PriceTable({"gpt-4o-mini": ModelPrice(input_per_mtok=0.15, output_per_mtok=0.60)}),
+        )
+    ],
+    checkpointer=InMemorySaver(),  # needed for pauses
+)
+```
+
+With Jev (needs `TYPESAFE_API_KEY`):
+
+```python
+from typesafe_sdk import TypeSafeClient
+from evenkeel import EvenKeelMiddleware, JevClassifier
+
+EvenKeelMiddleware(classifier=JevClassifier(TypeSafeClient()))
+```
+
+## How failures are handled
+
+| Kind | Example | Action |
+|---|---|---|
+| `transient` | 429, 503, timeout, dropped connection | Back off and retry the same call |
+| `bad_output` | Malformed output, a call that breaks the schema (recognised by Jev, or raised as `BadOutputError`) | Retry with a hint saying what was wrong |
+| `bad_input` | Input a tool rejects (`BadInputError`, HTTP 400 or 422, a pydantic `ValidationError`) | Model call: fail. Tool call: return the error to the model so it fixes its arguments |
+| `needs_human` | Refund over a limit, a policy block | Pause with `interrupt()` and wait for a decision |
+| `fatal` | A bug, bad credentials | Re-raise the original error at once |
+
+Retries and escalations stop at `max_attempts` (default 4). Each failure is recorded (error, kind, confidence,
+which classifier decided, action): in the agent's state under `evenkeel_attempts` when the call ends in a
+result, and on the raised error as `error.evenkeel_attempts` when it finally fails.
+
+Some failures never reach the middleware, because LangChain turns them into messages for the model itself:
+a call to a tool that does not exist, and tool arguments that fail the tool's schema.
+
+## Pauses and how to resume
+
+A failure that needs a person:
+
+```python
+from langgraph.types import Command
+
+agent.invoke(Command(resume={"decision": "retry", "hint": "approved by finance"}), config)
+agent.invoke(Command(resume={"decision": "fail"}), config)
+agent.invoke(Command(resume={"decision": "skip", "message": "Refunded by hand."}), config)  # tool calls only
+```
+
+A `retry` with a `hint` on a tool call sends the reviewer's words to the model, which can call the tool again; a
+tool cannot take a hint itself. A call that still needs a person after one approved retry fails. An answer
+that is not one of these values does not raise: the run pauses again, with an `error` field in the payload
+saying what was wrong.
+
+Over budget:
+
+```python
+agent.invoke(Command(resume={"approve": True, "budget_usd": 1.00}), config)
+agent.invoke(Command(resume={"approve": False}), config)  # ends the run
+```
+
+The budget applies to each run: every new `invoke` on a thread starts with zero spend and the configured
+budget.
+
+## Limitations
+
+- **Budgets can overshoot by one call.** The check runs before a model call, but a call's cost is known only
+  after it. keel has the same limitation.
+- **No built-in prices.** You supply them; a model without a price, or a response without token usage, is an
+  error rather than a silent $0. Dated or versioned names (`gpt-4o-mini-2024-07-18`, `claude-3-5-sonnet-20241022`, `gpt-4-0613`, `...-latest`)
+  use the price of their base name; any other variant (`gpt-4o-mini` against `gpt-4`, `gpt-4-turbo`) needs its own
+  entry.
+- **On resume after an escalation, the failing call runs once more** before the decision applies, because
+  LangGraph re-runs the paused node. For a tool with side effects, make it idempotent or use `skip`.
+- **Classification is only as good as the classifier.** Rules only recognise explicit signals (status codes,
+  network errors, evenkeel's error classes): on keel's 75 error messages from public SDK issues they pick the
+  right kind 29 times and the right action 39 times. Jev reads the message itself; see
+  [keel's results](https://github.com/yodablocks/keel#failure-classification-rules-vs-jev) for its accuracy on
+  real error messages, and `evals/` to measure it here.
+
+## Develop
+
+    uv sync --all-extras
+    uv run pytest

evenkeel-0.1.0/evenkeel/__init__.py

@@ -0,0 +1,34 @@
+"""Failure classification, retries with hints, dollar budgets and escalation for LangChain agents."""
+
+from ._core import AttemptRecord
+from .budget import ModelPrice, PriceTable
+from .failure import (
+    BadInputError,
+    BadOutputError,
+    FailureContext,
+    FailureVerdict,
+    NeedsHumanError,
+    OverBudgetError,
+    RuleClassifier,
+    default_policy,
+)
+from .jev import JevClassifier
+from .middleware import EvenKeelMiddleware
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "AttemptRecord",
+    "BadInputError",
+    "BadOutputError",
+    "EvenKeelMiddleware",
+    "FailureContext",
+    "FailureVerdict",
+    "JevClassifier",
+    "ModelPrice",
+    "NeedsHumanError",
+    "OverBudgetError",
+    "PriceTable",
+    "RuleClassifier",
+    "default_policy",
+]

evenkeel-0.1.0/evenkeel/_compat.py

@@ -0,0 +1,21 @@
+"""The one place that imports LangChain and LangGraph internals. If an import path moves in a new release,
+change it here only."""
+
+from langchain.agents.middleware import AgentMiddleware, AgentState, ModelRequest, ModelResponse, hook_config
+from langchain.agents.middleware.types import ExtendedModelResponse
+from langgraph.errors import GraphBubbleUp
+from langgraph.prebuilt.tool_node import ToolCallRequest
+from langgraph.types import Command, interrupt
+
+__all__ = [
+    "AgentMiddleware",
+    "AgentState",
+    "Command",
+    "ExtendedModelResponse",
+    "GraphBubbleUp",
+    "ModelRequest",
+    "ModelResponse",
+    "ToolCallRequest",
+    "hook_config",
+    "interrupt",
+]

evenkeel-0.1.0/evenkeel/_core.py

@@ -0,0 +1,126 @@
+"""Decision logic shared by every integration: apply the policy, enforce the attempt limit, build records,
+and read reviewers' resume values. No LangChain imports, so other integrations can reuse it."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Literal
+
+from typing_extensions import TypedDict
+
+from .failure import (
+    Escalate,
+    Fail,
+    FailureAction,
+    FailureContext,
+    FailurePolicy,
+    FailureVerdict,
+    Retry,
+    RetryWithHint,
+    error_message,
+)
+
+_RECORD_ERROR_CHARS = 500
+
+
+class AttemptRecord(TypedDict):
+    attempt: int
+    call: str
+    error_type: str
+    error: str
+    kind: str
+    confidence: float
+    source: str
+    action: str
+
+
+@dataclass(frozen=True)
+class Decision:
+    action: FailureAction
+    record: AttemptRecord
+
+
+def decide(ctx: FailureContext, verdict: FailureVerdict, policy: FailurePolicy) -> Decision:
+    action = policy(verdict, ctx)
+    if isinstance(action, (Retry, RetryWithHint, Escalate)) and ctx.attempt >= ctx.max_attempts:
+        action = Fail(reason=f"attempt limit reached ({ctx.max_attempts}): {error_message(ctx.error)}")
+    record = AttemptRecord(
+        attempt=ctx.attempt,
+        call=ctx.step or "",
+        error_type=type(ctx.error).__name__,
+        error=error_message(ctx.error)[:_RECORD_ERROR_CHARS],
+        kind=verdict.kind,
+        confidence=round(verdict.confidence, 3),
+        source=verdict.source,
+        action=action.type,
+    )
+    return Decision(action=action, record=record)
+
+
+def escalation_payload(ctx: FailureContext, verdict: FailureVerdict, reason: str) -> dict[str, Any]:
+    is_tool = (ctx.step or "").startswith("tool:")
+    return {
+        "type": "evenkeel_escalation",
+        "call": ctx.step or "",
+        "error_type": type(ctx.error).__name__,
+        "error": error_message(ctx.error)[:_RECORD_ERROR_CHARS],
+        "kind": verdict.kind,
+        "confidence": round(verdict.confidence, 3),
+        "reason": reason,
+        "accepted": ["retry", "fail", "skip"] if is_tool else ["retry", "fail"],
+    }
+
+
+@dataclass(frozen=True)
+class Resume:
+    decision: Literal["retry", "fail", "skip"]
+    hint: str | None
+    message: str | None
+
+
+_RESUME_HELP = (
+    'Resume with {"decision": "retry"} (optionally with "hint": "..."), {"decision": "fail"}, '
+    'or, for tool calls only, {"decision": "skip", "message": "..."}'
+)
+
+
+def parse_resume(value: Any, *, call_type: Literal["model", "tool"]) -> Resume:
+    if not isinstance(value, dict) or value.get("decision") not in ("retry", "fail", "skip"):
+        raise ValueError(f"Unrecognised escalation decision {value!r}. {_RESUME_HELP}")
+    decision = value["decision"]
+    hint = value.get("hint")
+    message = value.get("message")
+    if hint is not None and not isinstance(hint, str):
+        raise ValueError(f"The decision's hint must be a string, got {hint!r}. {_RESUME_HELP}")
+    if decision == "skip":
+        if call_type == "model":
+            raise ValueError(f"A model call cannot be skipped: it has no stand-in result. {_RESUME_HELP}")
+        if not isinstance(message, str):
+            raise ValueError(f"A skip decision needs a message for the model. {_RESUME_HELP}")
+        return Resume("skip", None, message)
+    return Resume(decision, hint, None)
+
+
+_BUDGET_HELP = 'Resume with {"approve": true, "budget_usd": <more than spent>} or {"approve": false}'
+
+
+def parse_budget_resume(value: Any, *, spent_usd: float) -> float | None:
+    if not isinstance(value, dict) or not isinstance(value.get("approve"), bool):
+        raise ValueError(f"Unrecognised budget decision {value!r}. {_BUDGET_HELP}")
+    if not value["approve"]:
+        return None
+    budget = value.get("budget_usd")
+    if not isinstance(budget, (int, float)) or isinstance(budget, bool) or budget <= spent_usd:
+        raise ValueError(
+            f"An approval needs a budget_usd above the ${spent_usd:.4f} already spent, "
+            f"got {budget!r}. {_BUDGET_HELP}"
+        )
+    return float(budget)
+
+
+def attach_attempts(error: BaseException, records: list[AttemptRecord]) -> None:
+    try:
+        error.evenkeel_attempts = list(records)  # type: ignore[attr-defined]
+    except AttributeError:  # some built-in exceptions refuse new attributes
+        if hasattr(error, "add_note"):  # Python 3.11+
+            error.add_note(f"evenkeel attempts: {records}")

evenkeel-0.1.0/evenkeel/budget.py

@@ -0,0 +1,74 @@
+"""Turns a model response's token usage into dollars, from prices the user supplies."""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+from langchain_core.messages import AIMessage
+
+
+class UnknownModelPriceError(ValueError):
+    """The response's model is not in the price table."""
+
+
+class MissingUsageError(ValueError):
+    """The response reported no token usage, so its cost is unknown."""
+
+
+@dataclass(frozen=True)
+class ModelPrice:
+    input_per_mtok: float
+    """Dollars per million input tokens."""
+    output_per_mtok: float
+    """Dollars per million output tokens."""
+
+
+def model_name_of(message: AIMessage) -> str | None:
+    metadata = message.response_metadata or {}
+    name = metadata.get("model_name") or metadata.get("model")
+    return name if isinstance(name, str) else None
+
+
+# What may follow a priced name: only a date or version number (-2024-07-18, -20241022, -0613, -001) or the
+# -latest alias. Anything else (-5-sonnet, -turbo, -1106-preview) is a different model with its own price.
+_VERSION_SUFFIX = re.compile(r"-(\d{4}-\d{2}-\d{2}|\d{3,8}|latest)")
+
+
+class PriceTable:
+    """Prices per model name. A name with a version or date suffix uses its base price
+    (gpt-4o-mini-2024-07-18 -> gpt-4o-mini); a different model never borrows another's price
+    (gpt-4o-mini does not use gpt-4, and gpt-4-turbo does not use gpt-4)."""
+
+    def __init__(self, prices: Mapping[str, ModelPrice]) -> None:
+        self._prices = dict(prices)
+
+    def price_for(self, model_name: str) -> ModelPrice:
+        if model_name in self._prices:
+            return self._prices[model_name]
+        matches = [
+            key
+            for key in self._prices
+            if model_name.startswith(key) and _VERSION_SUFFIX.fullmatch(model_name[len(key) :])
+        ]
+        if not matches:
+            raise UnknownModelPriceError(
+                f"No price for model {model_name!r}. Add it to PriceTable; known: {sorted(self._prices)}"
+            )
+        return self._prices[max(matches, key=len)]
+
+    def cost_usd(self, message: AIMessage) -> float:
+        name = model_name_of(message)
+        if name is None:
+            raise UnknownModelPriceError(
+                "The response does not say which model produced it; response_metadata keys: "
+                f"{sorted((message.response_metadata or {}).keys())}"
+            )
+        price = self.price_for(name)
+        usage = message.usage_metadata
+        if not usage:
+            raise MissingUsageError(f"Model {name!r} reported no token usage, so this call's cost is unknown")
+        return (
+            usage["input_tokens"] * price.input_per_mtok + usage["output_tokens"] * price.output_per_mtok
+        ) / 1e6