python-fastllm 0.0.36__tar.gz → 0.0.38__tar.gz

python_fastllm-0.0.38/PKG-INFO

@@ -0,0 +1,408 @@
+Metadata-Version: 2.4
+Name: python-fastllm
+Version: 0.0.38
+Author-email: Kerem Turgutlu <keremturgutlu@gmail.com>
+License: Apache-2.0
+Project-URL: Repository, https://github.com/AnswerDotAI/fastllm
+Keywords: nbdev
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: fastcore>=2.1.18
+Requires-Dist: aidialog>=0.0.8
+Requires-Dist: fastspec>=0.0.11
+Requires-Dist: pillow
+
+# fastllm
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+## Install
+
+Clone and install locally into your `aai-ws` env
+
+## Setup
+
+``` python
+from fastllm.types import Completion
+from aidialog.msg_parts import Msg, Part, Text, Thinking, ToolUse, InputImage, mk_tool_res_msg
+from fastllm.acomplete import acomplete
+import asyncio, json
+
+# Helpers
+def user(text): return Msg(role='user', content=[Text(text)])
+
+async def stream(msgs, model, max_think=10, **kw):
+    "Stream a response, printing each item's `formatted` (🧠 per thinking delta). Returns the final Completion."
+    cnt = 0
+    async for o in await acomplete(msgs, model, stream=True, **kw):
+        if not isinstance(o, Part): continue
+        if isinstance(o, Thinking):
+            cnt += 1
+            if cnt > max_think: continue
+        print(o.formatted, end='', flush=True)
+    print()
+    return o
+```
+
+``` python
+mtok = 1024
+```
+
+## Chat — One Interface, Every Provider
+
+The same `acomplete` call works with Claude, GPT, Gemini, and Kimi. Just change the model name:
+
+``` python
+models = [
+    ('claude-sonnet-4-20250514', {}),
+    ('gpt-4o-mini', {}),
+    ('models/gemini-3-flash-preview', {}),
+    ('accounts/fireworks/models/kimi-k2p5', dict(vendor_name='fireworks_ai'))
+]
+for name, kw in models:
+    r = await acomplete([user("Say 'hello' in French.")], model=name, max_tokens=mtok, **kw)
+    print(f"{name:>30s} → {r.message.content[0].text.strip()}")
+```
+
+          claude-sonnet-4-20250514 → Bonjour!
+                       gpt-4o-mini → In French, "hello" is said as "bonjour."
+     models/gemini-3-flash-preview → Bonjour.
+    accounts/fireworks/models/kimi-k2p5 → The user is asking me to say "hello" in French. This is a very straightforward request. The common ways to say hello in French are:
+
+    1. "Bonjour" - the standard, formal way to say hello/good day
+    2. "Salut" - informal way to say hi/hello (also used for goodbye)
+    3. "Bonsoir" - good evening
+    4. "Coucou" - very informal, cute way to say hi
+
+    Since the user just asked for "hello" without specifying context, "Bonjour" is the most appropriate and standard answer. I should provide the most common translation and perhaps mention the informal alternative for completeness.
+
+    The response should be simple and direct.
+
+## Multi-Turn — Swap Providers Mid-Conversation
+
+Build a conversation with one provider, then seamlessly continue it with another. `fastllm` translates between every provider’s native format automatically:
+
+``` python
+# Turn 1: Claude starts the conversation
+msgs = [user("Name the 3 largest planets in our solar system. One sentence.")]
+print("Claude: ", end='')
+r1 = await stream(msgs, model='claude-sonnet-4-20250514', max_tokens=mtok)
+```
+
+    Claude: The three largest planets in our solar system are Jupiter, Saturn, and Neptune.
+
+``` python
+# Turn 2: Switch to GPT — just change the model string
+msgs += [r1.message, user("Which one has the most moons?")]
+print("GPT: ", end='')
+r2 = await stream(msgs, model='gpt-4o-mini', max_tokens=mtok)
+```
+
+    GPT: As of now, Saturn has the most moons, with over 80 confirmed moons.
+
+``` python
+# Turn 3: Switch to Gemini — same msgs, different provider
+msgs += [r2.message, user("Summarize our conversation in one sentence.")]
+print("Gemini: ", end='')
+r3 = await stream(msgs, model='models/gemini-3-flash-preview', max_tokens=mtok)
+```
+
+    Gemini: The conversation identified the three largest planets in the solar system and noted that Saturn currently has the most moons.
+
+``` python
+# Turn 4: Switch to Kimi — works the same way
+msgs += [r3.message, user("Thanks! What's one surprising fact about Saturn?")]
+print("Kimi: ", end='')
+r4 = await stream(msgs, model='accounts/fireworks/models/kimi-k2p5', vendor_name='fireworks_ai', max_tokens=mtok)
+```
+
+    Kimi: 🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠Saturn is less dense than water, meaning it would theoretically float if you could find a bathtub large enough to hold it.
+
+## System Prompts
+
+Pass a system prompt to any provider — `fastllm` maps it to each API’s native mechanism (Anthropic `system`, OpenAI `instructions`, Gemini `system_instruction`):
+
+``` python
+sys = "You are a pirate chef. Always respond in pirate speak and mention food."
+
+print("Claude: ", end='')
+r = await stream([user("What should I do today?")], model='claude-sonnet-4-20250514', system=sys, max_tokens=mtok)
+
+print("Gemini: ", end='')
+r = await stream([user("What should I do today?")], model='models/gemini-3-flash-preview', system=sys, max_tokens=mtok)
+```
+
+    Claude: Ahoy there, me hearty! *tips chef's hat with a feathered plume* 
+
+    Ye be askin' what to do on this fine day, eh? Well, shiver me spatulas, I've got some tasty suggestions for ye!
+
+    First, ye should be raidin' yer galley (that be yer kitchen, landlubber!) and whip up some grub fit for a crew! Perhaps some hearty sea biscuits with honey, or a fine fish stew that'll warm yer bones like treasure warms the heart!
+
+    Then, if the weather be fair, take yerself outside and feel that salty breeze on yer face while ye munch on some portable provisions - maybe some dried fruits and nuts, perfect for any seafarin' adventure!
+
+    And if ye be feelin' ambitious, batten down the hatches and try cookin' somethin' new! A spicy jambalaya or some coconut rice that'll transport ye straight to the Caribbean seas!
+
+    Remember, me matey - a pirate's day ain't complete without good food in yer belly and the spirit of adventure in yer heart! Now off with ye, and may yer meals be as bountiful as buried treasure! 
+
+    *waves wooden spoon like a cutlass*
+
+    Arrr, what say ye? Does any of this grub sound temptin' to yer taste buds?
+    Gemini: Ahoy there, ye scurvy bilge-rat! If ye be lookin' for a way to spend yer daylight hours, I’ve got just the plan for a hungry soul like yerself.
+
+    First, ye should sharpen yer cutlass—not for fightin' off the Royal Navy, mind ye, but for slicin' through a thick slab of **salted pork** and some **ripe mangoes**! A dull blade is the mark of a lazy cook, and we'll have none of that on this vessel.
+
+    Once yer steel is keen, head down to the shoreline and see if ye can't scavenge some **clams** or a **fat crustacean** from the tide pools. If ye find a crab the size of a cannonball, we'll toss it in the pot with some stolen spices and a splash of grog to make a **bisque** that’ll make yer toes curl!
+
+    And if the wind be light, ye can spend the afternoon scrubbin' the barnacles off the hull—it builds an appetite for a massive bowl of **hardtack and lobscouse stew**. 
+
+    Now, quit yer lollygaggin' and get to the galley! There be **onions** that won't peel themselves, and I'll be hornswoggled if I let a single clove of **garlic** go to waste! Arrr!
+
+## Tool Calling — Define Once, Use Anywhere
+
+Define tools in a single canonical format. `fastllm` translates to each provider’s native tool schema automatically. Here we start a tool-use conversation with Claude, provide the result, then switch to GPT and Gemini to continue:
+
+``` python
+tools = [{"type": "function", "function": {
+    "name": "get_weather",
+    "description": "Get current weather for a city",
+    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
+}}]
+
+# Turn 1: Claude requests the tool
+msgs = [user("What's the weather in Paris?")]
+r1 = await stream(msgs, model='claude-sonnet-4-20250514', tools=tools, max_tokens=mtok)
+print("Tool calls:", r1.tool_calls)
+```
+
+    I'll check the current weather in Paris for you.
+    Tool calls: [ToolCall(id='toolu_01RMN1WM7vPBT3ovv5Ex6VzC', name='get_weather', arguments={'city': 'Paris'}, server=False, extra={'caller': {'type': 'direct'}})]
+
+``` python
+# Provide the tool result
+msgs += [r1.message, mk_tool_res_msg(r1.tool_calls, ['22°C, sunny with light clouds'])]
+
+# Turn 2: Switch to GPT to interpret the result
+msgs.append(user("Should I bring a jacket?"))
+print("GPT: ", end='')
+r2 = await stream(msgs, model='gpt-4o-mini', tools=tools, max_tokens=mtok)
+```
+
+    GPT: With the temperature at 22°C and sunny with light clouds, a jacket isn't necessary for most people. However, if you tend to feel cold easily or if you plan to be out in the evening when it might cool down a bit, it could be a good idea to bring a light jacket.
+
+``` python
+# Turn 3: Gemini sees the full cross-provider tool history
+msgs += [r2.message, user("How about tomorrow — will it rain?")]
+r3 = await stream(msgs, model='models/gemini-3-flash-preview', tools=tools, max_tokens=mtok, web_search_options={})
+```
+
+    Tomorrow in Paris (Saturday, June 13), the weather is expected to be mostly pleasant and sunny. 
+
+    While there is a small chance of light, passing showers (around a 20-25% chance), most forecasts indicate a dry day with mostly clear or scattered clouds. Temperatures will likely reach a comfortable high of around 22°C to 27°C (72°F to 80°F), making it a great day for being outdoors. 
+
+    So, while you might see a stray shower, you likely won't need to worry about heavy rain!
+
+## Tool Choice
+
+Control whether the model must use tools, can’t use tools, or decides on its own:
+
+``` python
+# Force the model to call a tool (even for a greeting)
+r = await stream([user("Hello there!")], model='claude-sonnet-4-20250514', tools=tools, tool_choice='required', max_tokens=mtok)
+print("Forced:", r.tool_calls)
+
+# Prevent tool use (model must answer directly)
+print("\nNo tools: ", end='')
+r = await stream([user("What's the weather?")], model='claude-sonnet-4-20250514', tools=tools, tool_choice='none', max_tokens=mtok)
+```
+
+
+    Forced: [ToolCall(id='toolu_01U7tAXjXAtwjp6xNSMbsyPU', name='get_weather', arguments={'city': '<UNKNOWN>'}, server=False, extra={'caller': {'type': 'direct'}})]
+
+    No tools: I'd be happy to help you get the weather information! However, I need to know which city you'd like me to check the weather for. Could you please tell me the city name?
+
+## Thinking / Extended Reasoning
+
+Enable model reasoning with `reasoning_effort`. The canonical values (`low`, `medium`, `high`) map to each provider’s native budget system. Thinking tokens stream as 🧠:
+
+``` python
+# Claude with thinking
+print("Claude: ", end='')
+r = await stream([user("What is 127 × 849?")], model='claude-sonnet-4-6', reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"\n🧠 {p.text[:150]}...")
+
+# Kimi with thinking — same interface
+print("\nKimi: ", end='')
+r = await stream([user("What is 127 × 849?")], model='accounts/fireworks/models/kimi-k2p5', vendor_name='fireworks_ai', reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if isinstance(p, Thinking): print(f"\n🧠 {p.text[:150]}...")
+```
+
+    Claude: 🧠🧠🧠## Calculating 127 × 849
+
+    **Breaking it down:**
+    - 127 × 800 = 101,600
+    - 127 × 40 = 5,080
+    - 127 × 9 = 1,143
+
+    **Adding the parts:**
+    101,600 + 5,080 + 1,143 = **107,823**
+
+    🧠 127 × 849:
+    127 × 800 = 101,600
+    127 × 49 = 127 × 50 - 127 = 6350 - 127 = 6223
+    Total: 107,823...
+
+    Kimi: 🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠**107,823**
+
+    Here's the calculation:
+    - 127 × 800 = 101,600
+    - 127 × 40 = 5,080  
+    - 127 × 9 = 1,143
+    - **Total: 101,600 + 5,080 + 1,143 = 107,823**
+
+    🧠 The user is asking for the product of 127 and 849. I need to calculate 127 × 849.
+
+    Let me calculate this step by step.
+
+    Method 1: Standard multiplicat...
+
+## Web Search (Server Tools)
+
+OpenAI’s Responses API supports server-side web search. Server tool calls are normalized alongside regular tool calls:
+
+``` python
+ws_tools = [{"type": "web_search_preview"}]
+print("GPT + web search: ", end='')
+r = await stream([user("What is the latest Python release?")], model='gpt-4o-mini', tools=ws_tools, max_tokens=512)
+print(f"\nServer tools used: {[tc.name for tc in r.tool_calls if tc.server]}")
+```
+
+    GPT + web search: As of June 12, 2026, the latest stable release of Python is version 3.14.5, which was released on May 10, 2026. ([test.python.org](https://test.python.org/downloads/latest?utm_source=openai))
+
+    Python 3.14 introduced several significant features, including:
+
+    - **PEP 779**: Official support for free-threaded Python, allowing threads to run more concurrently.
+    - **PEP 649**: Deferred evaluation of annotations, improving the semantics of using annotations.
+    - **PEP 750**: Template string literals (t-strings) for custom string processing, using the familiar syntax of f-strings.
+    - **PEP 734**: Support for multiple interpreters in the standard library.
+    - **PEP 784**: A new module `compression.zstd` providing support for the Zstandard compression algorithm.
+
+    For a comprehensive list of changes and improvements in Python 3.14, you can refer to the official release notes. ([docs.python.org](https://docs.python.org/it/3.14/whatsnew/3.14.html?utm_source=openai))
+
+    You can download the latest version of Python from the official Python website. ([python.org](https://www.python.org/downloads/?lang=python&utm_source=openai)) 
+
+    Server tools used: ['web_search']
+
+## Caching (Anthropic)
+
+Anthropic supports prompt caching via `cache_control`. Set it on a part’s `cache_control` — repeat calls with the same cached content save tokens:
+
+``` python
+# Cache a long system prompt (must be >1024 tokens for Anthropic caching)
+long_ctx = "You are an expert on the solar system. " * 200
+system = Text(long_ctx, cache_control={'type': 'ephemeral'})
+
+print("Call 1: ", end='')
+r1 = await stream([user("What is Jupiter's mass?")], model='claude-sonnet-4-20250514', system=system, max_tokens=mtok)
+print(f"Usage: {r1.usage}")
+
+print("Call 2: ", end='')
+r2 = await stream([user("How about Saturn?")], model='claude-sonnet-4-20250514', system=system, max_tokens=mtok)
+print(f"Usage: {r2.usage}")
+```
+
+    Call 1: Jupiter's mass is approximately 1.898 × 10²⁷ kilograms (or 1,898,000,000,000,000,000,000,000,000 kg).
+
+    To put this in perspective:
+    - Jupiter is about 318 times more massive than Earth
+    - It contains more than twice the mass of all other planets in our solar system combined
+    - Jupiter's mass is about 1/1047th the mass of the Sun
+
+    This enormous mass gives Jupiter its strong gravitational influence, which helps it act as a "cosmic vacuum cleaner" by capturing asteroids and comets that might otherwise threaten the inner planets.
+    Usage: Usage(prompt_tokens=1814, completion_tokens=145, total_tokens=1959, cached_tokens=0, cache_creation_tokens=1802, reasoning_tokens=0, raw={'input_tokens': 12, 'cache_creation_input_tokens': 1802, 'cache_read_input_tokens': 0, 'output_tokens': 145})
+    Call 2: Saturn is truly one of the most spectacular planets in our solar system! Here are some key facts about this gas giant:
+
+    **Basic Characteristics:**
+    - Sixth planet from the Sun
+    - Second largest planet (after Jupiter)
+    - Composed primarily of hydrogen and helium
+    - Has the lowest density of any planet - it would actually float in water!
+
+    **Famous Ring System:**
+    - Most prominent and extensive ring system in the solar system
+    - Made primarily of ice particles and rocky debris
+    - Rings are organized into distinct sections (A, B, C rings are the main ones)
+    - Ring particles range from tiny ice crystals to house-sized chunks
+
+    **Moons:**
+    - Has 146 confirmed moons (the most of any planet)
+    - Titan is its largest moon - larger than Mercury and has a thick atmosphere
+    - Enceladus has geysers of water ice erupting from its south pole
+    - Many other fascinating moons like Iapetus, Mimas, and Dione
+
+    **Physical Features:**
+    - Takes about 29.5 Earth years to orbit the Sun
+    - A day on Saturn is only about 10.7 hours
+    - Has extreme winds reaching up to 1,800 km/h
+    - Beautiful hexagonal storm at its north pole
+
+    **Exploration:**
+    - Visited by Pioneer 11, Voyager 1 & 2, and most notably the Cassini mission (2004-2017)
+
+    Is there a particular aspect of Saturn you'd like to know more about?
+    Usage: Usage(prompt_tokens=1812, completion_tokens=337, total_tokens=2149, cached_tokens=1802, cache_creation_tokens=0, reasoning_tokens=0, raw={'input_tokens': 10, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 1802, 'output_tokens': 337})
+
+## Media Inputs
+
+Send images to any provider that supports them. The canonical `input_image` part works everywhere:
+
+``` python
+img_url = "https://img.freepik.com/free-photo/mountain-range-body-water_53876-139760.jpg?semt=ais_hybrid&w=740&q=80"
+img_msg = Msg(role='user', content=[
+    InputImage(img_url),
+    Text("What do you see in this image?")
+])
+
+for name, kw in [('claude-sonnet-4-20250514', {}), ('gpt-4o-mini', {}), ('models/gemini-3-flash-preview', {})]:
+    print(f"{name:>30s}: ", end='')
+    r = await stream([img_msg], model=name, max_tokens=mtok, **kw)
+```
+
+          claude-sonnet-4-20250514: I see a beautiful, serene landscape photograph taken from what appears to be a wooden dock or platform. The image shows:
+
+    - A perfectly still lake that creates mirror-like reflections
+    - Snow-capped mountains in the background under a partly cloudy sky
+    - Dense forests of evergreen trees (likely pine or fir) lining the shoreline
+    - The mountains, trees, and sky all reflected clearly in the calm water
+    - Wooden planks in the foreground, suggesting the photo was taken from a dock or wooden viewing platform
+    - The lighting appears soft and atmospheric, possibly during golden hour
+    - The overall scene has a very peaceful, pristine wilderness quality typical of places like the Canadian Rockies, Alaska, or similar mountainous lake regions
+
+    The composition creates a sense of depth and tranquility, with the wooden platform in the foreground leading the eye toward the expansive natural landscape beyond.
+                       gpt-4o-mini: The image features a serene landscape with a body of water reflecting surrounding trees and mountains. In the foreground, there is a wooden deck or platform, enhancing the picturesque view. The scene is calm and depicts natural beauty, characterized by lush greenery, a clear sky, and distant snow-capped mountains.
+     models/gemini-3-flash-preview: This image depicts a serene and majestic landscape, viewed from the perspective of someone standing on a wooden deck.
+
+    Here is a breakdown of what is visible:
+
+    *   **Foreground:** In the immediate foreground are weathered brown wooden planks of a deck or pier. The planks run diagonally, drawing the eye toward the center of the image.
+    *   **Midground (The Lake):** Beyond the deck is a large, still lake. The water is a deep blue and acts as a perfect mirror, reflecting the trees and mountains above with startling clarity.
+    *   **Shoreline:** Along the far edge of the lake is a dense line of lush green evergreen trees. A thin, bright yellow or golden strip of vegetation runs along the very edge of the water.
+    *   **Background (Mountains):** Towering over the entire scene are massive, rugged mountains. The peaks are covered in white snow or glaciers and appear in shades of light blue and grey due to a soft, atmospheric haze.
+    *   **Sky:** The sky is a pale, bright blue, almost white in some areas, with a few soft clouds visible in the upper right corner.
+
+    The overall mood of the image is peaceful, cool-toned, and tranquil.
+
+`fastllm` supports four media part classes. Provider support varies:
+
+| Media part   | Anthropic | OpenAI Responses | OpenAI Chat | Gemini |
+|--------------|-----------|------------------|-------------|--------|
+| `InputImage` | ✅        | ✅               | ✅          | ✅     |
+| `InputAudio` | ❌        | ❌ (coming soon) | ✅          | ✅     |
+| `InputVideo` | ❌        | ❌               | ❌          | ✅     |
+| `InputFile`  | ✅        | ✅               | ✅          | ✅     |
+
+All media parts accept either a URL or a base64 data URL as their `text`. Unsupported combinations raise a clear `ValueError`.