python-fastllm 0.0.57__tar.gz → 0.0.59__tar.gz

python_fastllm-0.0.59/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: python-fastllm
+Version: 0.0.59
+Author-email: Kerem Turgutlu <keremturgutlu@gmail.com>
+License: Apache-2.0
+Project-URL: Repository, https://github.com/AnswerDotAI/fastllm
+Keywords: nbdev
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: fastcore>=2.1.18
+Requires-Dist: aidialog>=0.0.27
+Requires-Dist: fasttransport>=0.0.2
+Requires-Dist: pillow
+
+# fastllm
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+`fastllm` provides a common async interface for Anthropic, OpenAI Responses and Chat, Gemini, and OpenAI-compatible providers such as Kimi. Define messages and tools in a shared format and call `acomplete`. Select the model and, where needed, its provider without rewriting the conversation or tool loop. Supported features vary by provider.
+
+`AsyncResponses` provides a streaming Responses API interface over the same providers. It translates input items and events and keeps a canonical `ResponseState` between turns. For transports without provider continuation ids, it replays that state automatically.
+
+## Install
+
+Run `pip install python-fastllm` or clone from github and install locally.
+
+## Setup
+
+``` python
+from aidialog.msg_parts import Msg, Part, Text, Thinking, ToolUse, InputImage, mk_tool_res_msg, Completion
+from fastllm.acomplete import acomplete
+from fastllm.types import sample_img_url, sample_doms
+import asyncio, json
+```
+
+``` python
+def user(text): return Msg(role='user', content=[Text(text)])
+
+async def stream(msgs, model, max_think=0, **kw):
+    "Print streamed parts and return the final Completion; limit printed thinking deltas with max_think."
+    cnt = 0
+    async for o in await acomplete(msgs, model, stream=True, **kw):
+        if not isinstance(o, Part): continue
+        if isinstance(o, Thinking):
+            cnt += 1
+            if cnt > max_think: continue
+        print(o.formatted, end='', flush=True)
+    print()
+    return o
+```
+
+``` python
+mtok = 1024
+```
+
+## Chat across providers
+
+Use the same `acomplete` interface with Claude, GPT, Gemini, or Kimi. Set `model` and any provider-specific routing arguments:
+
+``` python
+models = [
+    ('claude-sonnet-5', {}),
+    ('gpt-4o-mini', {}),
+    ('models/gemini-3-flash-preview', {}),
+    ('accounts/fireworks/models/kimi-k3', dict(vendor_name='fireworks_ai'))
+]
+for name, kw in models:
+    r = await acomplete([user("Translate 'hello' into French. Return only the translation.")],
+                       model=name, max_tokens=mtok, **kw)
+    text = ''.join(p.text for p in r.message.content if isinstance(p, Text))
+    print(f"{name}: {text.strip()}")
+```
+
+    claude-sonnet-5: Bonjour
+    gpt-4o-mini: Bonjour
+    models/gemini-3-flash-preview: Bonjour
+    accounts/fireworks/models/kimi-k3: Bonjour
+
+## System prompts
+
+Pass `system` to supply a system prompt. `fastllm` translates it to the provider’s field: Anthropic `system`, OpenAI Responses `instructions`, or Gemini `system_instruction`.
+
+``` python
+sys = "You are a pirate chef. Always respond in pirate speak and mention food. Use one short sentence."
+
+print("Claude: ", end='')
+r = await stream([user("What should I do today?")], model='claude-sonnet-5', system=sys, max_tokens=mtok)
+
+print("Gemini: ", end='')
+r = await stream([user("What should I do today?")], model='models/gemini-3-flash-preview', system=sys, max_tokens=mtok)
+```
+
+    Claude: Arrr, chart a course fer the galley and cook up a hearty stew, matey!
+    Gemini: Sharpen yer cutlass and feast on spicy shark stew, ye scurvy dog!
+
+## Tool calling
+
+Define tools in the shared schema. `fastllm` translates it to the provider’s tool format. This example handles a tool request, adds the result to the history, and continues the conversation:
+
+``` python
+tools = [{"type": "function", "function": {
+    "name": "get_weather",
+    "description": "Get current weather for a city",
+    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
+}}]
+
+msgs = [user("What's the weather in Paris?")]
+r1 = await stream(msgs, model='claude-sonnet-5', tools=tools, max_tokens=mtok)
+print("Tool calls:", r1.tool_calls)
+```
+
+    - ⏳ `get_weather(city="Paris")` ⏳
+
+    Tool calls: [ToolUse(raw={'caller': {'type': 'direct'}}, cache_control=None, id='toolu_01XacBpJHnsjtPWJWr55nA2m', name='get_weather', arguments={'city': 'Paris'}, server=False, text=None)]
+
+``` python
+msgs += [r1.message, mk_tool_res_msg(r1.tool_calls, ['22°C, sunny with light clouds']),
+         user("Should I bring a jacket? Answer in one short sentence.")]
+r2 = await stream(msgs, model='claude-sonnet-5', tools=tools, max_tokens=mtok)
+```
+
+    No, a jacket isn't necessary — it's a mild, sunny 22°C day.
+
+## Tool Choice
+
+Control whether the model must use tools, can’t use tools, or decides on its own:
+
+``` python
+r = await acomplete([user("Hello there!")], model='claude-sonnet-5',
+                    tools=tools, tool_choice='required', max_tokens=mtok)
+print("Required:", [tc.name for tc in r.tool_calls])
+
+r = await acomplete([user("What's the weather?")], model='claude-sonnet-5',
+                    tools=tools, tool_choice='none', max_tokens=mtok)
+print("None:", r.tool_calls)
+```
+
+    Required: ['get_weather']
+    None: []
+
+## Thinking and extended reasoning
+
+Set `reasoning_effort` to `low`, `medium`, or `high` for supported models. The adapters translate these values to provider-specific settings. Reasoning output uses `Thinking` parts; this example prints a short excerpt alongside the answer.
+
+``` python
+print("Claude: ", end='')
+r = await stream([user("What is 127 × 849? Return just the number.")], model='claude-sonnet-4-6',
+                 reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"Thinking excerpt: {p.text[:150]}...")
+
+print("Kimi: ", end='')
+r = await stream([user("What is 127 × 849? Return just the number.")],
+                 model='accounts/fireworks/models/kimi-k3', vendor_name='fireworks_ai',
+                 reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"Thinking excerpt: {p.text[:150]}...")
+```
+
+    Claude: 107823
+    Thinking excerpt: 127 × 849 = 127 × 800 + 127 × 49 = 101600 + 6223 = 107823...
+    Kimi: 107823
+    Thinking excerpt: 127 × 849 = 107823. Let me verify: 127*800=101600; 127*49=6223; total 107823. Return just number....
+
+## Web Search (Server Tools)
+
+OpenAI’s Responses API supports server-side web search. Server tool calls are normalized alongside regular tool calls:
+
+``` python
+ws_tools = [{"type": "web_search_preview"}]
+r = await acomplete([user("What is the latest Python release? Reply with the version number only.")],
+                    model='gpt-4o-mini', tools=ws_tools, max_tokens=512)
+print("Server tools used:", [tc.name for tc in r.tool_calls if tc.server])
+```
+
+    Server tools used: ['web_search']
+
+## Caching with Anthropic
+
+Set a part’s `cache_control` to request prompt caching. Later requests can reuse eligible cached content. The example prints usage for two calls with the same system prompt:
+
+``` python
+long_ctx = "You are an expert on the solar system. " * 200
+system = Text(long_ctx, cache_control={'type': 'ephemeral'})
+
+r1 = await acomplete([user("What is Jupiter's mass? Answer in one sentence.")],
+                     model='claude-sonnet-5', system=system, max_tokens=mtok)
+print("First call: cache creation tokens", r1.usage.cache_creation_tokens)
+
+r2 = await acomplete([user("What is Saturn's mass? Answer in one sentence.")],
+                     model='claude-sonnet-5', system=system, max_tokens=mtok)
+print("Second call: cache read tokens", r2.usage.cached_tokens)
+```
+
+    First call: cache creation tokens 2204
+    Second call: cache read tokens 2204
+
+## Media inputs
+
+Use `InputImage` with a model that supports images. The example sends the same image message to three providers:
+
+``` python
+img_msg = Msg(role='user', content=[
+    InputImage(sample_img_url),
+    Text("List three visible objects, using nouns only.")
+])
+
+for name, kw in [('claude-sonnet-5', {}), ('gpt-4o-mini', {}), ('models/gemini-3-flash-preview', {})]:
+    print(f"{name}: ", end='')
+    r = await stream([img_msg], model=name, max_tokens=mtok, **kw)
+```
+
+    claude-sonnet-5: Mountains, lake, trees
+    gpt-4o-mini: 1. Mountains  
+    2. Trees  
+    3. Lake
+    models/gemini-3-flash-preview: planks, lake, mountains
+
+The media adapters support the following part types:
+
+| Media part   | Anthropic | OpenAI Responses | OpenAI Chat | Gemini |
+|--------------|-----------|------------------|-------------|--------|
+| `InputImage` | Yes       | Yes              | Yes         | Yes    |
+| `InputAudio` | No        | No               | Yes         | Yes    |
+| `InputVideo` | No        | No               | No          | Yes    |
+| `InputFile`  | Yes       | Yes              | Yes         | Yes    |
+
+Media parts accept a URL or base64 data URL as their `text`. An unsupported combination raises `ValueError`.

python_fastllm-0.0.59/README.md

@@ -0,0 +1,215 @@
+# fastllm
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+`fastllm` provides a common async interface for Anthropic, OpenAI Responses and Chat, Gemini, and OpenAI-compatible providers such as Kimi. Define messages and tools in a shared format and call `acomplete`. Select the model and, where needed, its provider without rewriting the conversation or tool loop. Supported features vary by provider.
+
+`AsyncResponses` provides a streaming Responses API interface over the same providers. It translates input items and events and keeps a canonical `ResponseState` between turns. For transports without provider continuation ids, it replays that state automatically.
+
+## Install
+
+Run `pip install python-fastllm` or clone from github and install locally.
+
+## Setup
+
+``` python
+from aidialog.msg_parts import Msg, Part, Text, Thinking, ToolUse, InputImage, mk_tool_res_msg, Completion
+from fastllm.acomplete import acomplete
+from fastllm.types import sample_img_url, sample_doms
+import asyncio, json
+```
+
+``` python
+def user(text): return Msg(role='user', content=[Text(text)])
+
+async def stream(msgs, model, max_think=0, **kw):
+    "Print streamed parts and return the final Completion; limit printed thinking deltas with max_think."
+    cnt = 0
+    async for o in await acomplete(msgs, model, stream=True, **kw):
+        if not isinstance(o, Part): continue
+        if isinstance(o, Thinking):
+            cnt += 1
+            if cnt > max_think: continue
+        print(o.formatted, end='', flush=True)
+    print()
+    return o
+```
+
+``` python
+mtok = 1024
+```
+
+## Chat across providers
+
+Use the same `acomplete` interface with Claude, GPT, Gemini, or Kimi. Set `model` and any provider-specific routing arguments:
+
+``` python
+models = [
+    ('claude-sonnet-5', {}),
+    ('gpt-4o-mini', {}),
+    ('models/gemini-3-flash-preview', {}),
+    ('accounts/fireworks/models/kimi-k3', dict(vendor_name='fireworks_ai'))
+]
+for name, kw in models:
+    r = await acomplete([user("Translate 'hello' into French. Return only the translation.")],
+                       model=name, max_tokens=mtok, **kw)
+    text = ''.join(p.text for p in r.message.content if isinstance(p, Text))
+    print(f"{name}: {text.strip()}")
+```
+
+    claude-sonnet-5: Bonjour
+    gpt-4o-mini: Bonjour
+    models/gemini-3-flash-preview: Bonjour
+    accounts/fireworks/models/kimi-k3: Bonjour
+
+## System prompts
+
+Pass `system` to supply a system prompt. `fastllm` translates it to the provider’s field: Anthropic `system`, OpenAI Responses `instructions`, or Gemini `system_instruction`.
+
+``` python
+sys = "You are a pirate chef. Always respond in pirate speak and mention food. Use one short sentence."
+
+print("Claude: ", end='')
+r = await stream([user("What should I do today?")], model='claude-sonnet-5', system=sys, max_tokens=mtok)
+
+print("Gemini: ", end='')
+r = await stream([user("What should I do today?")], model='models/gemini-3-flash-preview', system=sys, max_tokens=mtok)
+```
+
+    Claude: Arrr, chart a course fer the galley and cook up a hearty stew, matey!
+    Gemini: Sharpen yer cutlass and feast on spicy shark stew, ye scurvy dog!
+
+## Tool calling
+
+Define tools in the shared schema. `fastllm` translates it to the provider’s tool format. This example handles a tool request, adds the result to the history, and continues the conversation:
+
+``` python
+tools = [{"type": "function", "function": {
+    "name": "get_weather",
+    "description": "Get current weather for a city",
+    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
+}}]
+
+msgs = [user("What's the weather in Paris?")]
+r1 = await stream(msgs, model='claude-sonnet-5', tools=tools, max_tokens=mtok)
+print("Tool calls:", r1.tool_calls)
+```
+
+    - ⏳ `get_weather(city="Paris")` ⏳
+
+    Tool calls: [ToolUse(raw={'caller': {'type': 'direct'}}, cache_control=None, id='toolu_01XacBpJHnsjtPWJWr55nA2m', name='get_weather', arguments={'city': 'Paris'}, server=False, text=None)]
+
+``` python
+msgs += [r1.message, mk_tool_res_msg(r1.tool_calls, ['22°C, sunny with light clouds']),
+         user("Should I bring a jacket? Answer in one short sentence.")]
+r2 = await stream(msgs, model='claude-sonnet-5', tools=tools, max_tokens=mtok)
+```
+
+    No, a jacket isn't necessary — it's a mild, sunny 22°C day.
+
+## Tool Choice
+
+Control whether the model must use tools, can’t use tools, or decides on its own:
+
+``` python
+r = await acomplete([user("Hello there!")], model='claude-sonnet-5',
+                    tools=tools, tool_choice='required', max_tokens=mtok)
+print("Required:", [tc.name for tc in r.tool_calls])
+
+r = await acomplete([user("What's the weather?")], model='claude-sonnet-5',
+                    tools=tools, tool_choice='none', max_tokens=mtok)
+print("None:", r.tool_calls)
+```
+
+    Required: ['get_weather']
+    None: []
+
+## Thinking and extended reasoning
+
+Set `reasoning_effort` to `low`, `medium`, or `high` for supported models. The adapters translate these values to provider-specific settings. Reasoning output uses `Thinking` parts; this example prints a short excerpt alongside the answer.
+
+``` python
+print("Claude: ", end='')
+r = await stream([user("What is 127 × 849? Return just the number.")], model='claude-sonnet-4-6',
+                 reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"Thinking excerpt: {p.text[:150]}...")
+
+print("Kimi: ", end='')
+r = await stream([user("What is 127 × 849? Return just the number.")],
+                 model='accounts/fireworks/models/kimi-k3', vendor_name='fireworks_ai',
+                 reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"Thinking excerpt: {p.text[:150]}...")
+```
+
+    Claude: 107823
+    Thinking excerpt: 127 × 849 = 127 × 800 + 127 × 49 = 101600 + 6223 = 107823...
+    Kimi: 107823
+    Thinking excerpt: 127 × 849 = 107823. Let me verify: 127*800=101600; 127*49=6223; total 107823. Return just number....
+
+## Web Search (Server Tools)
+
+OpenAI’s Responses API supports server-side web search. Server tool calls are normalized alongside regular tool calls:
+
+``` python
+ws_tools = [{"type": "web_search_preview"}]
+r = await acomplete([user("What is the latest Python release? Reply with the version number only.")],
+                    model='gpt-4o-mini', tools=ws_tools, max_tokens=512)
+print("Server tools used:", [tc.name for tc in r.tool_calls if tc.server])
+```
+
+    Server tools used: ['web_search']
+
+## Caching with Anthropic
+
+Set a part’s `cache_control` to request prompt caching. Later requests can reuse eligible cached content. The example prints usage for two calls with the same system prompt:
+
+``` python
+long_ctx = "You are an expert on the solar system. " * 200
+system = Text(long_ctx, cache_control={'type': 'ephemeral'})
+
+r1 = await acomplete([user("What is Jupiter's mass? Answer in one sentence.")],
+                     model='claude-sonnet-5', system=system, max_tokens=mtok)
+print("First call: cache creation tokens", r1.usage.cache_creation_tokens)
+
+r2 = await acomplete([user("What is Saturn's mass? Answer in one sentence.")],
+                     model='claude-sonnet-5', system=system, max_tokens=mtok)
+print("Second call: cache read tokens", r2.usage.cached_tokens)
+```
+
+    First call: cache creation tokens 2204
+    Second call: cache read tokens 2204
+
+## Media inputs
+
+Use `InputImage` with a model that supports images. The example sends the same image message to three providers:
+
+``` python
+img_msg = Msg(role='user', content=[
+    InputImage(sample_img_url),
+    Text("List three visible objects, using nouns only.")
+])
+
+for name, kw in [('claude-sonnet-5', {}), ('gpt-4o-mini', {}), ('models/gemini-3-flash-preview', {})]:
+    print(f"{name}: ", end='')
+    r = await stream([img_msg], model=name, max_tokens=mtok, **kw)
+```
+
+    claude-sonnet-5: Mountains, lake, trees
+    gpt-4o-mini: 1. Mountains  
+    2. Trees  
+    3. Lake
+    models/gemini-3-flash-preview: planks, lake, mountains
+
+The media adapters support the following part types:
+
+| Media part   | Anthropic | OpenAI Responses | OpenAI Chat | Gemini |
+|--------------|-----------|------------------|-------------|--------|
+| `InputImage` | Yes       | Yes              | Yes         | Yes    |
+| `InputAudio` | No        | No               | Yes         | Yes    |
+| `InputVideo` | No        | No               | No          | Yes    |
+| `InputFile`  | Yes       | Yes              | Yes         | Yes    |
+
+Media parts accept a URL or base64 data URL as their `text`. An unsupported combination raises `ValueError`.

{python_fastllm-0.0.57 → python_fastllm-0.0.59}/fastllm/__init__.py

@@ -2,4 +2,4 @@
 
 - `fastllm.responses`: Provider-independent Responses resources and streaming over FastLLM"""
 
-__version__ = "0.0.57"
+__version__ = "0.0.59"

{python_fastllm-0.0.57 → python_fastllm-0.0.59}/fastllm/acomplete.py

@@ -39,7 +39,7 @@ vendor_mapping = {
     "fireworks_ai": ('openai_chat', "https://api.fireworks.ai/inference/v1", "FIREWORKS_API_KEY"),
     "qwen":         ('openai_chat', "https://dashscope.aliyuncs.com/compatible-mode/v1", "QWEN_API_KEY"),
     "minimax":      ('anthropic', "https://api.minimax.io/anthropic", "MINIMAX_API_KEY"),
-    "meta":         ('openai', "https://api.meta.ai/v1", "META_API_KEY"),
+    "meta":         ('openai_chat', "https://api.meta.ai/v1", "META_API_KEY"),
     "zai":          ('openai_chat', "https://api.z.ai/api/paas/v4", "ZAI_API_KEY")}
 
 # %% ../nbs/06_acomplete.ipynb #e3ed40fb

{python_fastllm-0.0.57 → python_fastllm-0.0.59}/fastllm/openai_responses.py

@@ -282,6 +282,7 @@ def cost(usage, m):
 # %% ../nbs/02_oai_responses.ipynb #2efda2a1
 def fix_payload(payload, model, vendor_name):
     "Apply vendor request quirks to a built payload."
+    if model == 'gpt-6-astra': payload.pop('temperature', None)
     if vendor_name == 'codex':
         for k in 'temperature max_tokens max_output_tokens max_completion_tokens metadata'.split(): payload.pop(k, None)
         payload['store'] = False

{python_fastllm-0.0.57 → python_fastllm-0.0.59}/fastllm/types.py

@@ -7,7 +7,7 @@ __all__ = ['FinishReason', 'api_registry', 'model_prices_url', 'sample_img_url',
            'sonn', 'sonn5', 'opus46', 'opus48', 'opus', 'opus5', 'fable', 'fable5', 'fable51', 'gpt54', 'gpt54m',
            'gpt55', 'codex54', 'codex54m', 'codex55', 'codex53spark', 'model_info_registry', 'modern_llm',
            'effort_codes', 'deepseek_v4_common', 'deepseek_v4_flash_prices', 'mimo_v25_common', 'codex_pricing', 'sol',
-           'terra', 'luna', 'gpt56s', 'Usage', 'APIRegistry', 'mk_completion', 'fn_schema', 'payload_kwargs',
+           'terra', 'luna', 'gpt56s', 'astra', 'Usage', 'APIRegistry', 'mk_completion', 'fn_schema', 'payload_kwargs',
            'provider_req', 'get_api_key', 'wrap_typed', 'unwrap_typed', 'resize_b64', 'model_prices_meta',
            'infer_api_name', 'get_model_meta', 'register_model_info', 'get_model_info', 'effort_code', 'effort_levels',
            'resolve_effort', 'get_model_pricing', 'approx_pricing', 'is_deepseek_peak_hour', 'price_tier', 'tier_rate']
@@ -317,11 +317,16 @@ for model in (haik45, sonn45, sonn46, sonn5, opus46, opus48, opus5, fable5, fabl
 
 # %% ../nbs/00_types.ipynb #bb0c4c2a
 sol,terra,luna = gpt56s = 'gpt-5.6-sol gpt-5.6-terra gpt-5.6-luna'.split()
+astra = 'gpt-6-astra'
+for model in (*gpt56s, astra): register_model_info(model, 'openai', base=model, max_input_tokens=272_000)
+
 # Codex serves only the suffixed names; bare `gpt-5.6` is rejected with a ChatGPT account.
-# Its window is smaller than the API's: 371,331 input tokens is accepted and 371,981 is not, on all three.
+# Its window accepts 371,331 input tokens and rejects 371,981, on all three.
 for model in gpt56s:
     register_model_info(model, 'codex', base=model, base_vendor_name='openai', max_input_tokens=371_000, **codex_pricing)
     _flat_rates('codex', model)
+register_model_info(astra, 'codex', base=astra, base_vendor_name='openai', **codex_pricing)
+_flat_rates('codex', astra)
 
 # %% ../nbs/00_types.ipynb #24cc47ec
 def get_model_pricing(mn, vendor_name, million=True):