anyray-agent 0.1.0__tar.gz

anyray_agent-0.1.0/.gitignore

@@ -0,0 +1,2 @@
+.coverage
+dist/

anyray_agent-0.1.0/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.5
+Name: anyray-agent
+Version: 0.1.0
+Summary: Keeps an agent answering when its Anyray gateway is unreachable: an httpx2 transport for the OpenAI, Anthropic and LangChain SDKs that resends a call straight to the provider only when the gateway never handled it.
+Project-URL: Homepage, https://docs.anyray.ai/enroll/agent-fallback-reference
+License-Expression: MIT
+Keywords: ai-gateway,anthropic,anyray,bedrock,fallback,langchain,llm,openai
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: httpx2<3,>=2.12
+Provides-Extra: httpx
+Requires-Dist: httpx<1,>=0.27; extra == 'httpx'
+Description-Content-Type: text/markdown
+
+# anyray-agent
+
+Keeps an agent answering when its [Anyray](https://anyray.ai) gateway is unreachable. Give your SDK an `httpx2`
+client built on `AnyrayTransport()`. While the gateway is up, every call goes through it unchanged. When the gateway
+is down, a call goes straight to the provider instead.
+
+```sh
+pip install anyray-agent
+```
+
+Python 3.11 or later. One dependency, `httpx2`, which the current OpenAI and Anthropic SDKs already use.
+
+## Use it
+
+Keep the client pointed at your gateway, with your agent key, exactly as before. Add one argument.
+
+```python
+import os
+
+import httpx2
+from anthropic import Anthropic
+from anyray_agent import AnyrayTransport
+
+client = Anthropic(
+    base_url=os.environ["ANYRAY_GATEWAY_URL"],
+    api_key=os.environ["ANYRAY_CLIENT_KEY"],
+    http_client=httpx2.Client(transport=AnyrayTransport()),
+)
+```
+
+| Client | Where `AnyrayTransport()` goes |
+| --- | --- |
+| OpenAI SDK | `OpenAI(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| Anthropic SDK | `Anthropic(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| LangChain `ChatOpenAI` | `ChatOpenAI(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| LangChain `ChatAnthropic` | `use_fallback(ChatAnthropic(...), transport=AnyrayTransport())` |
+
+`use_fallback` comes from `anyray_agent.langchain`: `ChatAnthropic` has no client argument, so it swaps the clients
+the model builds. Async code uses `AsyncAnyrayTransport` with `httpx2.AsyncClient` (`http_async_client=` on
+`ChatOpenAI`, `async_transport=` on `use_fallback`). An SDK still on `httpx` 0.x (openai < 3, anthropic < 2) takes
+the same classes from `anyray_agent.httpx` (`pip install "anyray-agent[httpx]"`).
+
+## The provider key
+
+A call can only go direct with a provider key on the host: `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are read from the
+environment, or pass them:
+
+```python
+AnyrayTransport(provider_keys={"openai": "…", "anthropic": "…"})
+```
+
+To send Anthropic calls to AWS Bedrock instead of api.anthropic.com, name the region. Credentials come from
+`AWS_BEARER_TOKEN_BEDROCK` or the standard AWS variables, or pass `api_key` or `credentials`. For an IAM role or a
+profile, pass boto3's credential chain:
+
+```python
+import boto3
+
+AnyrayTransport(bedrock={"region": "us-east-1", "credentials": boto3.Session().get_credentials})
+```
+
+Without a key for a provider, its calls are never sent direct.
+
+## When a call goes direct
+
+- **Sent direct:** the connection to the gateway never opened (refused, DNS, connect timeout) and the gateway's
+  `/livez` check also fails. After the outage has lasted your org's fallback time (120 seconds by default), calls go
+  direct without trying the gateway first. They return once the gateway has answered for your org's recover time
+  (10 minutes by default).
+- **Never sent direct:** a call that may already have reached the provider (a reset mid-request, any 5xx), so
+  nothing is billed twice. Also a model or route only the gateway serves, and any call while your org has turned
+  fallback off. Nothing goes direct until the agent has read that setting from the gateway.
+
+The gateway key is never sent to a provider.
+
+## See what it is doing
+
+```python
+transport = AnyrayTransport(on_status=lambda status: print(status.state))
+transport.status()  # FallbackStatus(state="armed" | "direct" | "blocked" | "off", reason=..., since=..., calls_direct=...)
+```
+
+The agent also reports its state to the gateway, without any request content, so it shows on the Agents page.
+
+Full reference: [docs.anyray.ai/enroll/agent-fallback-reference](https://docs.anyray.ai/enroll/agent-fallback-reference).
+
+MIT licensed.

anyray_agent-0.1.0/README.md

@@ -0,0 +1,87 @@
+# anyray-agent
+
+Keeps an agent answering when its [Anyray](https://anyray.ai) gateway is unreachable. Give your SDK an `httpx2`
+client built on `AnyrayTransport()`. While the gateway is up, every call goes through it unchanged. When the gateway
+is down, a call goes straight to the provider instead.
+
+```sh
+pip install anyray-agent
+```
+
+Python 3.11 or later. One dependency, `httpx2`, which the current OpenAI and Anthropic SDKs already use.
+
+## Use it
+
+Keep the client pointed at your gateway, with your agent key, exactly as before. Add one argument.
+
+```python
+import os
+
+import httpx2
+from anthropic import Anthropic
+from anyray_agent import AnyrayTransport
+
+client = Anthropic(
+    base_url=os.environ["ANYRAY_GATEWAY_URL"],
+    api_key=os.environ["ANYRAY_CLIENT_KEY"],
+    http_client=httpx2.Client(transport=AnyrayTransport()),
+)
+```
+
+| Client | Where `AnyrayTransport()` goes |
+| --- | --- |
+| OpenAI SDK | `OpenAI(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| Anthropic SDK | `Anthropic(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| LangChain `ChatOpenAI` | `ChatOpenAI(..., http_client=httpx2.Client(transport=AnyrayTransport()))` |
+| LangChain `ChatAnthropic` | `use_fallback(ChatAnthropic(...), transport=AnyrayTransport())` |
+
+`use_fallback` comes from `anyray_agent.langchain`: `ChatAnthropic` has no client argument, so it swaps the clients
+the model builds. Async code uses `AsyncAnyrayTransport` with `httpx2.AsyncClient` (`http_async_client=` on
+`ChatOpenAI`, `async_transport=` on `use_fallback`). An SDK still on `httpx` 0.x (openai < 3, anthropic < 2) takes
+the same classes from `anyray_agent.httpx` (`pip install "anyray-agent[httpx]"`).
+
+## The provider key
+
+A call can only go direct with a provider key on the host: `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are read from the
+environment, or pass them:
+
+```python
+AnyrayTransport(provider_keys={"openai": "…", "anthropic": "…"})
+```
+
+To send Anthropic calls to AWS Bedrock instead of api.anthropic.com, name the region. Credentials come from
+`AWS_BEARER_TOKEN_BEDROCK` or the standard AWS variables, or pass `api_key` or `credentials`. For an IAM role or a
+profile, pass boto3's credential chain:
+
+```python
+import boto3
+
+AnyrayTransport(bedrock={"region": "us-east-1", "credentials": boto3.Session().get_credentials})
+```
+
+Without a key for a provider, its calls are never sent direct.
+
+## When a call goes direct
+
+- **Sent direct:** the connection to the gateway never opened (refused, DNS, connect timeout) and the gateway's
+  `/livez` check also fails. After the outage has lasted your org's fallback time (120 seconds by default), calls go
+  direct without trying the gateway first. They return once the gateway has answered for your org's recover time
+  (10 minutes by default).
+- **Never sent direct:** a call that may already have reached the provider (a reset mid-request, any 5xx), so
+  nothing is billed twice. Also a model or route only the gateway serves, and any call while your org has turned
+  fallback off. Nothing goes direct until the agent has read that setting from the gateway.
+
+The gateway key is never sent to a provider.
+
+## See what it is doing
+
+```python
+transport = AnyrayTransport(on_status=lambda status: print(status.state))
+transport.status()  # FallbackStatus(state="armed" | "direct" | "blocked" | "off", reason=..., since=..., calls_direct=...)
+```
+
+The agent also reports its state to the gateway, without any request content, so it shows on the Agents page.
+
+Full reference: [docs.anyray.ai/enroll/agent-fallback-reference](https://docs.anyray.ai/enroll/agent-fallback-reference).
+
+MIT licensed.

anyray_agent-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "anyray-agent"
+version = "0.1.0"
+description = "Keeps an agent answering when its Anyray gateway is unreachable: an httpx2 transport for the OpenAI, Anthropic and LangChain SDKs that resends a call straight to the provider only when the gateway never handled it."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.11"
+dependencies = ["httpx2>=2.12,<3"]
+keywords = ["anyray", "ai-gateway", "llm", "fallback", "openai", "anthropic", "bedrock", "langchain"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Typing :: Typed",
+]
+
+[project.optional-dependencies]
+# For SDKs still on httpx 0.x (openai < 3, anthropic < 2): `from anyray_agent.httpx import AnyrayTransport`.
+httpx = ["httpx>=0.27,<1"]
+
+[project.urls]
+Homepage = "https://docs.anyray.ai/enroll/agent-fallback-reference"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/anyray_agent"]
+
+[tool.hatch.build.targets.sdist]
+# Hatchling also packs the nearest .gitignore whatever this says: the package keeps its own so the monorepo
+# root's never ships, and scripts/checkDist.py refuses any other.
+only-include = ["src/anyray_agent", "README.md", "pyproject.toml"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+# Tests import the package from src/, so CI needs no build backend or install step.
+pythonpath = ["src"]
+
+[tool.coverage.run]
+branch = true
+source_pkgs = ["anyray_agent"]
+
+[tool.coverage.report]
+# Measured 95% (lines + branches) when the port landed; the untested lines are close-time and defensive edges.
+fail_under = 94
+show_missing = true

anyray_agent-0.1.0/src/anyray_agent/__init__.py

@@ -0,0 +1,41 @@
+"""Automatic direct-provider fallback for Anyray gateway clients using httpx2."""
+
+from collections.abc import Callable, Mapping
+from typing import cast
+
+import httpx2
+
+from ._core import AsyncCore, SyncCore
+from ._types import (FALLBACK_LEVELS, AsyncTransport, BedrockOptions, FallbackStatus,
+                     Library, Request, SyncTransport)
+
+__version__ = "0.1.0"
+
+
+class AnyrayTransport(SyncCore, httpx2.BaseTransport):
+    """Wrap a sync transport; caller-supplied inner transports remain caller-owned."""
+
+    def __init__(self, *, provider_keys: Mapping[str, str] | None = None,
+                 bedrock: BedrockOptions | None = None, transport: httpx2.BaseTransport | None = None,
+                 on_status: Callable[[FallbackStatus], None] | None = None) -> None:
+        super().__init__(cast(Library, httpx2), provider_keys=provider_keys, bedrock=bedrock,
+                         transport=cast(SyncTransport | None, transport), on_status=on_status)
+
+    def handle_request(self, request: httpx2.Request) -> httpx2.Response:
+        return cast(httpx2.Response, self.run(self.send(cast(Request, request))))
+
+
+class AsyncAnyrayTransport(AsyncCore, httpx2.AsyncBaseTransport):
+    """Wrap an async transport and cancel its background work on aclose()."""
+
+    def __init__(self, *, provider_keys: Mapping[str, str] | None = None,
+                 bedrock: BedrockOptions | None = None, transport: httpx2.AsyncBaseTransport | None = None,
+                 on_status: Callable[[FallbackStatus], None] | None = None) -> None:
+        super().__init__(cast(Library, httpx2), provider_keys=provider_keys, bedrock=bedrock,
+                         transport=cast(AsyncTransport | None, transport), on_status=on_status)
+
+    async def handle_async_request(self, request: httpx2.Request) -> httpx2.Response:
+        return cast(httpx2.Response, await self.run(self.send(cast(Request, request))))
+
+
+__all__ = ["AnyrayTransport", "AsyncAnyrayTransport", "FallbackStatus", "FALLBACK_LEVELS", "__version__"]

anyray_agent-0.1.0/src/anyray_agent/_bedrock.py

@@ -0,0 +1,168 @@
+"""Bedrock request rewriting, model resolution and SigV4 signing."""
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import os
+import re
+import threading
+from collections.abc import Mapping
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from typing import cast
+from urllib.parse import quote, urlsplit
+
+from . import _runtime
+from ._io import Close, Resolve, Send, Work
+from ._rules import compact, header, provider_key
+from ._types import BedrockOptions, CredentialSource, Library, Request, Response
+
+RESOLVED_MODELS: dict[str, str] = {}
+_CACHE_LOCK = threading.Lock()
+BEDROCK_ID = re.compile(r"^([a-z-]+\.)?anthropic\.|^arn:aws[a-z-]*:bedrock:")
+
+
+def is_bedrock_model(model: str) -> bool:
+    return model.startswith("claude-") or bool(BEDROCK_ID.search(model))
+
+
+@dataclass(frozen=True, repr=False)
+class Keys:
+    access: str
+    secret: str
+    token: str | None
+
+
+def normalize(value: object) -> Keys | None:
+    frozen = getattr(value, "get_frozen_credentials", None)
+    if callable(frozen):
+        value = frozen()
+    if isinstance(value, Mapping):
+        access, secret, token = (value.get(name) for name in ("access_key_id", "secret_access_key", "session_token"))
+    else:
+        access, secret, token = (getattr(value, name, None) for name in ("access_key", "secret_key", "token"))
+    access = provider_key(access) if isinstance(access, str) else None
+    secret = provider_key(secret) if isinstance(secret, str) else None
+    raw_token = token.strip() if isinstance(token, str) else None
+    token = provider_key(raw_token)
+    if not access or not secret or (raw_token and not token):
+        return None
+    return Keys(access, secret, token)
+
+
+def region_prefix(region: str) -> str:
+    if region.startswith("us-gov-"):
+        return ""
+    for family, prefix in (("us-", "us."), ("eu-", "eu."), ("ap-", "apac."), ("ca-", "ca.")):
+        if region.startswith(family):
+            return prefix
+    return ""
+
+
+def sign(url: str, body: bytes, headers: dict[str, str], keys: Keys, region: str) -> None:
+    amz_date = datetime.fromtimestamp(_runtime.now(), timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+    date = amz_date[:8]
+    headers["x-amz-date"] = amz_date
+    if keys.token:
+        headers["x-amz-security-token"] = keys.token
+    parts = urlsplit(url)
+    canonical_headers = {name: value for name, value in headers.items() if name != "accept"}
+    canonical_headers["host"] = parts.netloc
+    names = sorted(canonical_headers)
+    signed = ";".join(names)
+    values = {name: re.sub(r"\s+", " ", canonical_headers[name].strip()) for name in names}
+    text = "".join(f"{name}:{values[name]}\n" for name in names)
+    uri = "/".join(quote(part, safe="-_.~") for part in parts.path.split("/"))
+    canonical = f"POST\n{uri}\n\n{text}\n{signed}\n{hashlib.sha256(body).hexdigest()}"
+    scope = f"{date}/{region}/bedrock/aws4_request"
+    to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashlib.sha256(canonical.encode()).hexdigest()}"
+    key = ("AWS4" + keys.secret).encode()
+    for value in (date, region, "bedrock", "aws4_request"):
+        key = hmac.digest(key, value.encode(), "sha256")
+    signature = hmac.new(key, to_sign.encode(), "sha256").hexdigest()
+    headers["authorization"] = (f"AWS4-HMAC-SHA256 Credential={keys.access}/{scope}, "
+                                f"SignedHeaders={signed}, Signature={signature}")
+
+
+@dataclass(repr=False)
+class Plan:
+    requests: list[Request]
+    cache_key: str | None = None
+    candidates: list[str] | None = None
+    streaming: bool = False
+
+    def execute(self) -> Work[Response]:
+        for index, request in enumerate(self.requests):
+            response = cast(Response, (yield Send(request)))
+            if response.status_code < 400 and self.cache_key and self.candidates:
+                with _CACHE_LOCK:
+                    RESOLVED_MODELS[self.cache_key] = self.candidates[index]
+            retry = response.status_code in (400, 404) and re.match(
+                r"^(ValidationException|ResourceNotFoundException)", header(response.headers, "x-amzn-errortype") or "")
+            if retry and index + 1 < len(self.requests):
+                yield Close(response)
+                continue
+            return response
+        raise RuntimeError("Empty direct plan")
+
+
+class Bedrock:
+    def __init__(self, options: BedrockOptions, *, defer_refresh: bool = False) -> None:
+        self.region = options["region"]
+        if not re.fullmatch(r"[a-z]{2}(-gov)?-[a-z]+-[0-9]+", self.region):
+            raise ValueError("Invalid Bedrock region")
+        self.api_key = provider_key(options.get("api_key")) or provider_key(os.environ.get("AWS_BEARER_TOKEN_BEDROCK"))
+        configured = options.get("credentials")
+        self.source: CredentialSource = configured if configured is not None else {
+            "access_key_id": os.environ.get("AWS_ACCESS_KEY_ID"),
+            "secret_access_key": os.environ.get("AWS_SECRET_ACCESS_KEY"),
+            "session_token": os.environ.get("AWS_SESSION_TOKEN"),
+        }
+        self.models = dict(options.get("models", {}))
+        try:
+            self.available = bool(self.api_key or callable(self.source)
+                                  or (defer_refresh and callable(getattr(self.source, "get_frozen_credentials", None)))
+                                  or normalize(self.source))
+        except Exception:
+            self.available = False
+
+    def plan(self, original: dict[str, object], incoming: Mapping[str, str], extensions: dict[str, object],
+             lib: Library) -> Work[Plan | str]:
+        keys = None
+        if not self.api_key:
+            try:
+                keys = cast(Keys | None, (yield Resolve(self.source)))
+            except Exception:
+                return "no_provider_key"
+            if keys is None:
+                return "no_provider_key"
+        model = cast(str, original["model"])
+        cache_key = f"{self.region}:{model}"
+        with _CACHE_LOCK:
+            known = self.models.get(model, RESOLVED_MODELS.get(cache_key))
+        base = f"{region_prefix(self.region)}anthropic.{model}"
+        candidates = [known] if known is not None else [model] if BEDROCK_ID.search(model) else [base, base + "-v1:0"]
+        body = {name: value for name, value in original.items()
+                if name not in ("model", "stream", "anthropic_version", "anthropic_beta")}
+        body["anthropic_version"] = "bedrock-2023-05-31"
+        betas = [beta.strip() for beta in (header(incoming, "anthropic-beta") or "").split(",") if beta.strip()]
+        if betas:
+            body["anthropic_beta"] = betas
+        content = compact(body).encode()
+        streaming = original.get("stream") is True
+        requests = []
+        for candidate in candidates:
+            # encodeURIComponent leaves these punctuation characters untouched; SigV4 encodes them again.
+            encoded = quote(candidate, safe="-_.!~*'()")
+            action = "invoke-with-response-stream" if streaming else "invoke"
+            url = f"https://bedrock-runtime.{self.region}.amazonaws.com/model/{encoded}/{action}"
+            headers = {"content-type": "application/json",
+                       "accept": "application/vnd.amazon.eventstream" if streaming else "application/json"}
+            if self.api_key:
+                headers["authorization"] = f"Bearer {self.api_key}"
+            elif keys:
+                sign(url, content, headers, keys, self.region)
+            requests.append(lib.Request("POST", url, headers=headers, content=content, extensions=extensions.copy()))
+        # Explicit mappings belong to this transport, never the process-wide cache.
+        return Plan(requests, None if model in self.models else cache_key, candidates, streaming)