chatlas 0.9.1__tar.gz → 0.10.0__tar.gz

{chatlas-0.9.1 → chatlas-0.10.0}/CHANGELOG.md

@@ -7,6 +7,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 -->
 
+## [0.10.0] - 2025-08-19
+
+### New features
+
+* Added `ChatCloudflare()` for chatting via [Cloudflare AI](https://developers.cloudflare.com/workers-ai/get-started/rest-api/). (#150) 
+* Added `ChatDeepSeek()` for chatting via [DeepSeek](https://www.deepseek.com/). (#147)
+* Added `ChatOpenRouter()` for chatting via [Open Router](https://openrouter.ai/). (#148)
+* Added `ChatHuggingFace()` for chatting via [Hugging Face](https://huggingface.co/). (#144)
+* Added `ChatMistral()` for chatting via [Mistral AI](https://mistral.ai/). (#145)
+* Added `ChatPortkey()` for chatting via [Portkey AI](https://portkey.ai/). (#143)
+
+### Changes
+
+* `ChatAnthropic()` and `ChatBedrockAnthropic()` now default to Claude Sonnet 4.0.
+
+### Bug fixes
+
+* Fixed an issue where chatting with some models was leading to `KeyError: 'cached_input'`. (#149)
+
+
+## [0.9.2] - 2025-08-08
+
+### Improvements
+
+* `Chat.get_cost()` now covers many more models and also takes cached tokens into account. (#133)
+* Avoid erroring when tool calls occur with recent versions of `openai` (> v1.99.5). (#141)
+
+
 ## [0.9.1] - 2025-07-09
 
 ### Bug fixes

chatlas-0.10.0/CLAUDE.md

@@ -0,0 +1,167 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Commands
+
+The project uses `uv` for package management and Make for common tasks:
+
+- **Setup environment**: `make setup` (installs all dependencies with `uv sync --all-extras`)
+- **Run tests**: `make check-tests` or `uv run pytest`
+- **Type checking**: `make check-types` or `uv run pyright`
+- **Linting/formatting**: `make check-format` (check) or `make format` (fix)
+- **Full checks**: `make check` (runs format, type, and test checks)
+- **Build package**: `make build` (creates dist/ with built package)
+- **Run single test**: `uv run pytest tests/test_specific_file.py::TestClass::test_method -v`
+- **Update snapshots**: `make update-snaps` (for syrupy snapshot tests)
+- **Documentation**: `make docs` (build) or `make docs-preview` (serve locally)
+
+## Project Architecture
+
+### Core Components
+
+**Chat System**: The main `Chat` class in `_chat.py` manages conversation state and provider interactions. It's a generic class that works with different providers through the `Provider` abstract base class.
+
+**Provider Pattern**: All LLM providers (OpenAI, Anthropic, Google, etc.) inherit from `Provider` in `_provider.py`. Each provider (e.g., `_provider_openai.py`) implements:
+- Model-specific parameter handling 
+- API client configuration
+- Request/response transformation
+- Tool calling integration
+
+**Content System**: The `_content.py` module defines structured content types:
+- `ContentText`: Plain text messages
+- `ContentImage`: Image content (inline, remote, or file-based)
+- `ContentToolRequest`/`ContentToolResult`: Tool interaction messages
+- `ContentJson`: Structured data responses
+
+**Tool System**: Tools are defined in `_tools.py` and allow LLMs to call Python functions. The system supports:
+- Function registration with automatic schema generation
+- Tool approval workflows
+- MCP (Model Context Protocol) server integration via `_mcp_manager.py`
+
+**Turn Management**: `Turn` objects in `_turn.py` represent individual conversation exchanges, containing sequences of `Content` objects.
+
+### Key Patterns
+
+1. **Provider Abstraction**: All providers implement the same interface but handle model-specific details internally
+2. **Generic Typing**: Heavy use of TypeVars and generics for type safety across providers
+3. **Streaming Support**: Both sync and async streaming responses via `ChatResponse`/`ChatResponseAsync`
+4. **Content-Based Messaging**: All communication uses structured `Content` objects rather than raw strings
+5. **Tool Integration**: Seamless function calling with automatic JSON schema generation from Python type hints
+
+### Testing Structure
+
+- Tests are organized by component (e.g., `test_provider_openai.py`, `test_tools.py`)
+- Snapshot testing with `syrupy` for response validation
+- MCP server tests use local test servers in `tests/mcp_servers/`
+- Async tests configured via `pytest.ini` with `asyncio_mode=strict`
+
+### Documentation
+
+Documentation is built with Quarto and quartodoc:
+- API reference generated from docstrings in `chatlas/` modules
+- Guides and examples in `docs/` as `.qmd` files
+- Type definitions in `chatlas/types/` provide provider-specific parameter types
+
+
+## Adding New Providers
+
+When implementing a new LLM provider, follow this systematic approach:
+
+### 1. Research Phase
+- **Check ellmer first**: Look in `../ellmer/R/provider-*.R` for existing implementations
+- **Identify base provider**: Most providers inherit from either `OpenAIProvider` (for OpenAI-compatible APIs) or implement `Provider` directly
+- **Check existing patterns**: Review similar providers in `chatlas/_provider_*.py`
+
+### 2. Implementation Steps
+1. **Create provider file**: `chatlas/_provider_[name].py`
+   - Use PascalCase for class names (e.g., `MistralProvider`)  
+   - Use snake_case for function names (e.g., `ChatMistral`)
+   - Follow existing docstring patterns with Prerequisites, Examples, Parameters, Returns sections
+   
+2. **Provider class structure**:
+   ```python
+   class [Name]Provider(OpenAIProvider):  # or Provider if custom
+       def __init__(self, ...):
+           super().__init__(...)
+           # Provider-specific initialization
+       
+       def _chat_perform_args(self, ...):
+           # Customize request parameters if needed
+           kwargs = super()._chat_perform_args(...)
+           # Apply provider-specific modifications
+           return kwargs
+   ```
+
+3. **Chat function signature**:
+   ```python
+   def Chat[Name](
+       *,
+       system_prompt: Optional[str] = None,
+       model: Optional[str] = None,
+       api_key: Optional[str] = None,
+       base_url: str = "https://...",
+       seed: int | None | MISSING_TYPE = MISSING,
+       kwargs: Optional["ChatClientArgs"] = None,
+   ) -> Chat["SubmitInputArgs", ChatCompletion]:
+   ```
+
+### 3. Testing Setup
+1. **Create test file**: `tests/test_provider_[name].py`
+2. **Add environment variable skip pattern**:
+   ```python
+   import os
+   import pytest
+   
+   do_test = os.getenv("TEST_[NAME]", "true")
+   if do_test.lower() == "false":
+       pytest.skip("Skipping [Name] tests", allow_module_level=True)
+   ```
+3. **Use standard test patterns**:
+   - `test_[name]_simple_request()`
+   - `test_[name]_simple_streaming_request()`  
+   - `test_[name]_respects_turns_interface()`
+   - `test_[name]_tool_variations()` (if supported)
+   - `test_data_extraction()`
+   - `test_[name]_images()` (if vision supported)
+
+### 4. Package Integration
+1. **Update `chatlas/__init__.py`**:
+   - Add import: `from ._provider_[name] import Chat[Name]`
+   - Add to `__all__` tuple: `"Chat[Name]"`
+
+2. **Run validation**:
+   ```bash
+   uv run pyright chatlas/_provider_[name].py
+   TEST_[NAME]=false uv run pytest tests/test_provider_[name].py -v
+   uv run python -c "from chatlas import Chat[Name]; print('Import successful')"
+   ```
+
+### 5. Provider-Specific Customizations
+
+**OpenAI-Compatible Providers**:
+- Inherit from `OpenAIProvider`
+- Override `_chat_perform_args()` for API differences
+- Common customizations: remove `stream_options`, adjust parameter names, modify headers
+
+**Custom API Providers**:
+- Inherit from `Provider` directly
+- Implement all abstract methods: `chat_perform()`, `chat_perform_async()`, `stream_text()`, etc.
+- Handle model-specific response formats
+
+### 6. Common Patterns
+- **Environment variables**: Use `[PROVIDER]_API_KEY` format
+- **Default models**: Use provider's recommended general-purpose model
+- **Seed handling**: `seed = 1014 if is_testing() else None` when MISSING
+- **Error handling**: Provider APIs often return different error formats
+- **Rate limiting**: Consider implementing client-side throttling for providers that need it
+
+### 7. Documentation Requirements
+- Include provider description and prerequisites
+- Document known limitations (tool calling, vision support, etc.)
+- Provide working examples with environment variable usage
+- Note any special model requirements (e.g., vision models for images)
+
+## Connections to ellmer
+
+This project is the Python equivalent of the R package ellmer. The source code for ellmer is available in a sibling directory to this project. Before implementing new features or bug fixes in chatlas, it may be useful to consult  the ellmer codebase to: (1) check whether the feature/fix already exists on the R side and (2) make sure the projects are aligned in terms of stylistic approaches. Note also that ellmer itself has a CLAUDE.md file which has a useful overview of the project. 

{chatlas-0.9.1 → chatlas-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: chatlas
-Version: 0.9.1
+Version: 0.10.0
 Summary: A simple and consistent interface for chatting with LLMs
 Project-URL: Homepage, https://posit-dev.github.io/chatlas
 Project-URL: Documentation, https://posit-dev.github.io/chatlas
@@ -69,7 +69,7 @@ Provides-Extra: ollama
 Provides-Extra: openai
 Provides-Extra: perplexity
 Provides-Extra: snowflake
-Requires-Dist: snowflake-ml-python; extra == 'snowflake'
+Requires-Dist: snowflake-ml-python<=1.9.0; extra == 'snowflake'
 Provides-Extra: test
 Requires-Dist: pyright>=1.1.379; extra == 'test'
 Requires-Dist: pytest-asyncio; extra == 'test'
@@ -79,7 +79,7 @@ Provides-Extra: vertex
 Requires-Dist: google-genai>=1.14.0; extra == 'vertex'
 Description-Content-Type: text/markdown
 
-# chatlas <a href="https://posit-dev.github.io/chatlas"><img src="docs/logos/hex/logo.png" align="right" height="138" alt="chatlas website" /></a>
+# chatlas <a href="https://posit-dev.github.io/chatlas"><img src="https://posit-dev.github.io/chatlas/logos/hex/logo.png" align="right" height="138" alt="chatlas website" /></a>
 
 <p>
 <!-- badges start -->
@@ -135,7 +135,7 @@ chat.chat("How's the weather in San Francisco?")
 ```
 
 
-<img src="docs/images/chatlas-hello.png" alt="Model response output to the user query: 'How's the weather in San Francisco?'" width="67%" style="display: block; margin-left: auto; margin-right: auto">
+<img src="https://posit-dev.github.io/chatlas/images/chatlas-hello.png" alt="Model response output to the user query: 'How's the weather in San Francisco?'" width="67%" style="display: block; margin-left: auto; margin-right: auto">
 
 
 Learn more at <https://posit-dev.github.io/chatlas>

{chatlas-0.9.1 → chatlas-0.10.0}/README.md

@@ -1,4 +1,4 @@
-# chatlas <a href="https://posit-dev.github.io/chatlas"><img src="docs/logos/hex/logo.png" align="right" height="138" alt="chatlas website" /></a>
+# chatlas <a href="https://posit-dev.github.io/chatlas"><img src="https://posit-dev.github.io/chatlas/logos/hex/logo.png" align="right" height="138" alt="chatlas website" /></a>
 
 <p>
 <!-- badges start -->
@@ -54,7 +54,7 @@ chat.chat("How's the weather in San Francisco?")
 ```
 
 
-<img src="docs/images/chatlas-hello.png" alt="Model response output to the user query: 'How's the weather in San Francisco?'" width="67%" style="display: block; margin-left: auto; margin-right: auto">
+<img src="https://posit-dev.github.io/chatlas/images/chatlas-hello.png" alt="Model response output to the user query: 'How's the weather in San Francisco?'" width="67%" style="display: block; margin-left: auto; margin-right: auto">
 
 
 Learn more at <https://posit-dev.github.io/chatlas>

{chatlas-0.9.1 → chatlas-0.10.0}/chatlas/__init__.py

@@ -1,20 +1,26 @@
 from . import types
-from ._anthropic import ChatAnthropic, ChatBedrockAnthropic
 from ._auto import ChatAuto
 from ._chat import Chat
 from ._content import ContentToolRequest, ContentToolResult, ContentToolResultImage
 from ._content_image import content_image_file, content_image_plot, content_image_url
 from ._content_pdf import content_pdf_file, content_pdf_url
-from ._databricks import ChatDatabricks
-from ._github import ChatGithub
-from ._google import ChatGoogle, ChatVertex
-from ._groq import ChatGroq
 from ._interpolate import interpolate, interpolate_file
-from ._ollama import ChatOllama
-from ._openai import ChatAzureOpenAI, ChatOpenAI
-from ._perplexity import ChatPerplexity
 from ._provider import Provider
-from ._snowflake import ChatSnowflake
+from ._provider_anthropic import ChatAnthropic, ChatBedrockAnthropic
+from ._provider_cloudflare import ChatCloudflare
+from ._provider_databricks import ChatDatabricks
+from ._provider_deepseek import ChatDeepSeek
+from ._provider_github import ChatGithub
+from ._provider_google import ChatGoogle, ChatVertex
+from ._provider_groq import ChatGroq
+from ._provider_huggingface import ChatHuggingFace
+from ._provider_mistral import ChatMistral
+from ._provider_ollama import ChatOllama
+from ._provider_openai import ChatAzureOpenAI, ChatOpenAI
+from ._provider_openrouter import ChatOpenRouter
+from ._provider_perplexity import ChatPerplexity
+from ._provider_portkey import ChatPortkey
+from ._provider_snowflake import ChatSnowflake
 from ._tokens import token_usage
 from ._tools import Tool, ToolRejectError
 from ._turn import Turn
@@ -28,14 +34,20 @@ __all__ = (
     "ChatAnthropic",
     "ChatAuto",
     "ChatBedrockAnthropic",
+    "ChatCloudflare",
     "ChatDatabricks",
+    "ChatDeepSeek",
     "ChatGithub",
     "ChatGoogle",
     "ChatGroq",
+    "ChatHuggingFace",
+    "ChatMistral",
     "ChatOllama",
     "ChatOpenAI",
+    "ChatOpenRouter",
     "ChatAzureOpenAI",
     "ChatPerplexity",
+    "ChatPortkey",
     "ChatSnowflake",
     "ChatVertex",
     "Chat",

{chatlas-0.9.1 → chatlas-0.10.0}/chatlas/_auto.py

@@ -5,16 +5,16 @@ from typing import Callable, Literal, Optional
 
 import orjson
 
-from ._anthropic import ChatAnthropic, ChatBedrockAnthropic
 from ._chat import Chat
-from ._databricks import ChatDatabricks
-from ._github import ChatGithub
-from ._google import ChatGoogle, ChatVertex
-from ._groq import ChatGroq
-from ._ollama import ChatOllama
-from ._openai import ChatAzureOpenAI, ChatOpenAI
-from ._perplexity import ChatPerplexity
-from ._snowflake import ChatSnowflake
+from ._provider_anthropic import ChatAnthropic, ChatBedrockAnthropic
+from ._provider_databricks import ChatDatabricks
+from ._provider_github import ChatGithub
+from ._provider_google import ChatGoogle, ChatVertex
+from ._provider_groq import ChatGroq
+from ._provider_ollama import ChatOllama
+from ._provider_openai import ChatAzureOpenAI, ChatOpenAI
+from ._provider_perplexity import ChatPerplexity
+from ._provider_snowflake import ChatSnowflake
 
 AutoProviders = Literal[
     "anthropic",

{chatlas-0.9.1 → chatlas-0.10.0}/chatlas/_chat.py

@@ -65,6 +65,7 @@ class TokensDict(TypedDict):
     role: Literal["user", "assistant"]
     tokens: int
     tokens_total: int
+    tokens_cached: int
 
 
 CompletionT = TypeVar("CompletionT")
@@ -293,12 +294,15 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
             {
                 "role": "user",
                 "tokens": turns[1].tokens[0],
+                # Number of tokens currently cached (reduces input token usage)
+                "tokens_cached": turns[1].tokens[2],
                 "tokens_total": turns[1].tokens[0],
             },
             # The token count for the 1st assistant response
             {
                 "role": "assistant",
                 "tokens": turns[1].tokens[1],
+                "tokens_cached": 0,
                 "tokens_total": turns[1].tokens[1],
             },
         ]
@@ -319,8 +323,11 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
                 [
                     {
                         "role": "user",
-                        # Implied token count for the user input
+                        # Implied new token count for the user input (input tokens - context - cached reads)
+                        # Cached reads are only subtracted for particular providers
                         "tokens": tj.tokens[0] - sum(ti.tokens),
+                        # Number of tokens currently cached (reduces input token usage depending on provider's API)
+                        "tokens_cached": tj.tokens[2],
                         # Total tokens = Total User Tokens for the Turn = Distinct new tokens + context sent
                         "tokens_total": tj.tokens[0],
                     },
@@ -329,6 +336,7 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
                         # The token count for the assistant response
                         "tokens": tj.tokens[1],
                         # Total tokens = Total Assistant tokens used in the turn
+                        "tokens_cached": 0,
                         "tokens_total": tj.tokens[1],
                     },
                 ]
@@ -339,7 +347,7 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
     def get_cost(
         self,
         options: Literal["all", "last"] = "all",
-        token_price: Optional[tuple[float, float]] = None,
+        token_price: Optional[tuple[float, float, float]] = None,
     ) -> float:
         """
         Estimate the cost of the chat.
@@ -357,11 +365,13 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
               - `"last"`: Return the cost of the last turn in the chat.
         token_price
             An optional tuple in the format of (input_token_cost,
-            output_token_cost) for bringing your own cost information.
+            output_token_cost, cached_token_cost) for bringing your own cost information.
                  - `"input_token_cost"`: The cost per user token in USD per
                    million tokens.
                  - `"output_token_cost"`: The cost per assistant token in USD
                    per million tokens.
+                - `"cached_token_cost"`: The cost per cached token read in USD
+                   per million tokens.
 
         Returns
         -------
@@ -374,15 +384,19 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
         if token_price:
             input_token_price = token_price[0] / 1e6
             output_token_price = token_price[1] / 1e6
+            cached_token_price = token_price[2] / 1e6
         else:
             price_token = get_token_pricing(self.provider.name, self.provider.model)
             if not price_token:
                 raise KeyError(
-                    f"We could not locate pricing information for model '{self.provider.model}' from provider '{self.provider.name}'. "
+                    f"We could not locate pricing information for model '{self.provider.model}'"
+                    f" from provider '{self.provider.name}'. "
                     "If you know the pricing for this model, specify it in `token_price`."
                 )
+
             input_token_price = price_token["input"] / 1e6
-            output_token_price = price_token["output"] / 1e6
+            output_token_price = price_token.get("output", 0) / 1e6
+            cached_token_price = price_token.get("cached_input", 0) / 1e6
 
         if len(turns_tokens) == 0:
             return 0.0
@@ -399,8 +413,16 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
             user_tokens = sum(
                 u["tokens_total"] for u in turns_tokens if u["role"] == "user"
             )
-            cost = (asst_tokens * output_token_price) + (
-                user_tokens * input_token_price
+            # We add the cached tokens here because for relevant providers they have already been subtracted
+            # from the user tokens. This assumes the provider uses (reads) the cache each time.
+            cached_token_reads = sum(
+                u["tokens_cached"] for u in turns_tokens if u["role"] == "user"
+            )
+
+            cost = (
+                (asst_tokens * output_token_price)
+                + (user_tokens * input_token_price)
+                + (cached_token_reads * cached_token_price)
             )
             return cost
 
@@ -408,7 +430,9 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
         if last_turn["role"] == "assistant":
             return last_turn["tokens"] * output_token_price
         if last_turn["role"] == "user":
-            return last_turn["tokens_total"] * input_token_price
+            return (last_turn["tokens_total"] * input_token_price) + (
+                last_turn["tokens_cached"] * cached_token_price
+            )
         raise ValueError(
             f"Expected last turn to have a role of 'user' or `'assistant'`, not '{last_turn['role']}'"
         )
@@ -2224,8 +2248,12 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
         tokens = self.get_tokens()
         tokens_asst = sum(u["tokens_total"] for u in tokens if u["role"] == "assistant")
         tokens_user = sum(u["tokens_total"] for u in tokens if u["role"] == "user")
+        tokens_cached = sum(u["tokens_cached"] for u in tokens if u["role"] == "user")
 
-        res = f"<Chat {self.provider.name}/{self.provider.model} turns={len(turns)} tokens={tokens_user}/{tokens_asst}"
+        res = (
+            f"<Chat {self.provider.name}/{self.provider.model} turns={len(turns)}"
+            f" tokens={tokens_user + tokens_cached}/{tokens_asst}"
+        )
 
         # Add cost info only if we can compute it
         cost = compute_cost(
@@ -2233,6 +2261,7 @@ class Chat(Generic[SubmitInputArgsT, CompletionT]):
             self.provider.model,
             tokens_user,
             tokens_asst,
+            tokens_cached,
         )
         if cost is not None:
             res += f" ${round(cost, ndigits=2)}"

chatlas-0.9.1/chatlas/_anthropic.py → chatlas-0.10.0/chatlas/_provider_anthropic.py

@@ -163,7 +163,7 @@ def ChatAnthropic(
     """
 
     if model is None:
-        model = log_model_default("claude-3-7-sonnet-latest")
+        model = log_model_default("claude-sonnet-4-0")
 
     return Chat(
         provider=AnthropicProvider(
@@ -586,7 +586,16 @@ class AnthropicProvider(
                         )
                     )
 
-        tokens = completion.usage.input_tokens, completion.usage.output_tokens
+        usage = completion.usage
+        # N.B. Currently, Anthropic doesn't cache by default and we currently do not support
+        # manual caching in chatlas. Note also that this only tracks reads, NOT writes, which
+        # have their own cost. To track that properly, we would need another caching category and per-token cost.
+
+        tokens = (
+            completion.usage.input_tokens,
+            completion.usage.output_tokens,
+            usage.cache_read_input_tokens if usage.cache_read_input_tokens else 0,
+        )
 
         tokens_log(self, tokens)
 
@@ -733,8 +742,7 @@ def ChatBedrockAnthropic(
     """
 
     if model is None:
-        # Default model from https://github.com/anthropics/anthropic-sdk-python?tab=readme-ov-file#aws-bedrock
-        model = log_model_default("anthropic.claude-3-5-sonnet-20241022-v2:0")
+        model = log_model_default("us.anthropic.claude-sonnet-4-20250514-v1:0")
 
     return Chat(
         provider=AnthropicBedrockProvider(
@@ -764,7 +772,7 @@ class AnthropicBedrockProvider(AnthropicProvider):
         aws_session_token: str | None,
         max_tokens: int = 4096,
         base_url: str | None,
-        name: str = "AnthropicBedrock",
+        name: str = "AWS/Bedrock",
         kwargs: Optional["ChatBedrockClientArgs"] = None,
     ):
         super().__init__(name=name, model=model, max_tokens=max_tokens)