python-fastllm 0.0.1__tar.gz

python_fastllm-0.0.1/PKG-INFO

@@ -0,0 +1,395 @@
+Metadata-Version: 2.4
+Name: python-fastllm
+Version: 0.0.1
+Author-email: Kerem Turgutlu <keremturgutlu@gmail.com>
+License: Apache-2.0
+Project-URL: Repository, https://github.com/AnswerDotAI/fastllm
+Keywords: nbdev
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: fastspec
+Requires-Dist: toolslm
+
+# fastllm
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+## Install
+
+Clone and install locally into your `aai-ws` env
+
+## Setup
+
+``` python
+from fastllm.types import Msg, Part, PartType, Completion
+from fastllm.acomplete import acomplete, mk_tool_res_msg
+import asyncio, json
+
+# Helpers
+def user(text): return Msg(role='user', content=[Part(type=PartType.text, text=text)])
+
+async def stream(msgs, model, **kw):
+    "Stream a response, printing text/thinking as it arrives. Returns the final Completion."
+    cnt, max_think = 0, 10
+    async for o in await acomplete(msgs, model, stream=True, **kw):
+        if not isinstance(o, Completion):
+            if o.get('thinking') and cnt < max_think: print('🧠', end='', flush=True)
+            if txt := o.get('text'): print(txt, end='', flush=True)
+            cnt += 1
+    print()
+    return o
+```
+
+``` python
+mtok = 1024
+```
+
+## Chat — One Interface, Every Provider
+
+The same [`acomplete`](./acomplete.html#acomplete) call works with
+Claude, GPT, Gemini, and Kimi. Just change the model name:
+
+``` python
+models = [
+    ('claude-sonnet-4-20250514', {}),
+    ('gpt-4o-mini', {}),
+    ('models/gemini-3-flash-preview', {}),
+    ('accounts/fireworks/models/kimi-k2p5', dict(vendor_name='fireworks_ai'))
+]
+for name, kw in models:
+    r = await acomplete([user("Say 'hello' in French.")], model=name, max_tokens=mtok, **kw)
+    print(f"{name:>30s} → {r.message.content[0].text.strip()}")
+```
+
+          claude-sonnet-4-20250514 → Bonjour!
+                       gpt-4o-mini → "Hello" in French is "Bonjour."
+     models/gemini-3-flash-preview → Bonjour.
+    accounts/fireworks/models/kimi-k2p5 → The user wants me to say "hello" in French. The word for "hello" in French is "Bonjour". 
+
+    This is a simple, straightforward request. I should provide the translation and perhaps a bit of context (like when it's used), but the main thing is to say "Bonjour".
+
+    I should not overcomplicate this. Just provide the French word for hello.
+
+## Multi-Turn — Swap Providers Mid-Conversation
+
+Build a conversation with one provider, then seamlessly continue it with
+another. `fastllm` translates between every provider’s native format
+automatically:
+
+``` python
+# Turn 1: Claude starts the conversation
+msgs = [user("Name the 3 largest planets in our solar system. One sentence.")]
+print("Claude: ", end='')
+r1 = await stream(msgs, model='claude-sonnet-4-20250514', max_tokens=mtok)
+```
+
+    Claude: The three largest planets in our solar system are Jupiter, Saturn, and Neptune.
+
+``` python
+# Turn 2: Switch to GPT — just change the model string
+msgs += [r1.message, user("Which one has the most moons?")]
+print("GPT: ", end='')
+r2 = await stream(msgs, model='gpt-4o-mini', max_tokens=mtok)
+```
+
+    GPT: As of now, Saturn has the most moons, surpassing Jupiter.
+
+``` python
+# Turn 3: Switch to Gemini — same msgs, different provider
+msgs += [r2.message, user("Summarize our conversation in one sentence.")]
+print("Gemini: ", end='')
+r3 = await stream(msgs, model='models/gemini-3-flash-preview', max_tokens=mtok)
+```
+
+    Gemini: We identified the largest planets in our solar system and confirmed that Saturn currently has the most moons.
+
+``` python
+# Turn 4: Switch to Kimi — works the same way
+msgs += [r3.message, user("Thanks! What's one surprising fact about Saturn?")]
+print("Kimi: ", end='')
+r4 = await stream(msgs, model='accounts/fireworks/models/kimi-k2p5', vendor_name='fireworks_ai', max_tokens=mtok)
+```
+
+    Kimi: 🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠Saturn is the only planet less dense than water, meaning it would theoretically float in a giant bathtub.
+
+## System Prompts
+
+Pass a system prompt to any provider — `fastllm` maps it to each API’s
+native mechanism (Anthropic `system`, OpenAI `instructions`, Gemini
+`system_instruction`):
+
+``` python
+sys = "You are a pirate chef. Always respond in pirate speak and mention food."
+
+print("Claude: ", end='')
+r = await stream([user("What should I do today?")], model='claude-sonnet-4-20250514', system=sys, max_tokens=mtok)
+
+print("Gemini: ", end='')
+r = await stream([user("What should I do today?")], model='models/gemini-3-flash-preview', system=sys, max_tokens=mtok)
+```
+
+    Claude: Ahoy there, me hearty! *tips pirate hat* 
+
+    Ye be askin' what to do today, eh? Well, as a seasoned sea cook, I'd say ye should start by raidin' yer galley for some fine grub! Perhaps whip up some hearty hardtack biscuits or a steamin' bowl of seafarer's stew with potatoes and salted pork, arrr!
+
+    If ye be feelin' adventurous, why not venture out to the market like ye be searchin' for buried treasure? Hunt down the finest fish, the ripest tropical fruits, and maybe some exotic spices from distant shores! 
+
+    And if the weather be fair, consider cookin' outdoors on the deck - er, I mean yer patio! Fire up the grill and roast some catch of the day with a side of ship's biscuits. Nothing beats the taste of food cooked under the open sky, savvy?
+
+    Whatever ye choose to do, make sure ye don't sail on an empty stomach, ye scallywag! A well-fed pirate be a happy pirate! 
+
+    *waves wooden spoon in the air* 
+
+    Now off with ye, and may yer day be filled with delicious adventures! Arrr! 🏴‍☠️
+    Gemini: Ahoy there, ye scurvy dog! If ye be lookin' for a way to spend the tide, I’ve got a recipe for adventure—and a belly full of grub!
+
+    First, haul yer carcass down to the galley! A true buccaneer starts the day by breakin' his fast with a bowl of **Salmagundi**—that be a fine mess of chopped meats, pickled herrings, and whatever greens we haven't fed to the parrots yet. 
+
+    Once yer belly is lined with **salted pork and hardtack**, ye ought to sharpen yer cutlass and go huntin' for the greatest treasure of all: a merchant ship carryin' a fresh crate of **cinnamon and exotic nutmeg**! 
+
+    If the winds be calm, ye can spend the afternoon fishin' off the stern for some **mahi-mahi**. We’ll grill it over a charcoal brazier with a squeeze of **sour lime** to keep the scurvy from claimin' yer teeth!
+
+    Now scurry off before I make ye peel a mountain of **onions** for me famous "Shipwreck Stew"! Arrr!
+
+## Tool Calling — Define Once, Use Anywhere
+
+Define tools in a single canonical format. `fastllm` translates to each
+provider’s native tool schema automatically. Here we start a tool-use
+conversation with Claude, provide the result, then switch to GPT and
+Gemini to continue:
+
+``` python
+tools = [{"type": "function", "function": {
+    "name": "get_weather",
+    "description": "Get current weather for a city",
+    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
+}}]
+
+# Turn 1: Claude requests the tool
+msgs = [user("What's the weather in Paris?")]
+r1 = await stream(msgs, model='claude-sonnet-4-20250514', tools=tools, max_tokens=mtok)
+print("Tool calls:", r1.tool_calls)
+```
+
+    I'll get the current weather information for Paris.
+    Tool calls: [ToolCall(id='toolu_01JNRNBQxzxT7uMXFQh16g8H', name='get_weather', arguments={'city': 'Paris'}, server=False, extra={'caller': {'type': 'direct'}})]
+
+``` python
+# Provide the tool result
+msgs += [r1.message, mk_tool_res_msg(r1.tool_calls, ['22°C, sunny with light clouds'])]
+
+# Turn 2: Switch to GPT to interpret the result
+msgs.append(user("Should I bring a jacket?"))
+print("GPT: ", end='')
+r2 = await stream(msgs, model='gpt-4o-mini', tools=tools, max_tokens=mtok)
+```
+
+    GPT: With the current temperature in Paris at 22°C and sunny with light clouds, you likely won't need a jacket. A light layer might be comfortable if you're out in the evening, but otherwise, it should be warm enough.
+
+``` python
+# Turn 3: Gemini sees the full cross-provider tool history
+msgs += [r2.message, user("How about tomorrow — will it rain?")]
+r3 = await stream(msgs, model='models/gemini-3-flash-preview', tools=tools, max_tokens=mtok, web_search_options={})
+```
+
+    No, it's not expected to rain in Paris tomorrow, Thursday, April 30. 
+
+    The forecast is looking very pleasant with sunny skies throughout the day and clear conditions at night. You can expect a high of around **24°C (75°F)** and a low of **11°C (52°F)**. It should be a great day for outdoor activities, though the temperature will drop in the evening, so you might want that jacket if you're out late! 
+
+    Rain is currently predicted to return on Friday, May 1.
+
+## Tool Choice
+
+Control whether the model must use tools, can’t use tools, or decides on
+its own:
+
+``` python
+# Force the model to call a tool (even for a greeting)
+r = await stream([user("Hello there!")], model='claude-sonnet-4-20250514', tools=tools, tool_choice='required', max_tokens=mtok)
+print("Forced:", r.tool_calls)
+
+# Prevent tool use (model must answer directly)
+print("\nNo tools: ", end='')
+r = await stream([user("What's the weather?")], model='claude-sonnet-4-20250514', tools=tools, tool_choice='none', max_tokens=mtok)
+```
+
+
+    Forced: [ToolCall(id='toolu_01PvVn1rxozZjW3FBwmVPHms', name='get_weather', arguments={'city': '<UNKNOWN>'}, server=False, extra={'caller': {'type': 'direct'}})]
+
+    No tools: I'd be happy to help you check the weather! However, I need to know which city you'd like me to check the weather for. Could you please tell me the name of the city?
+
+## Thinking / Extended Reasoning
+
+Enable model reasoning with `reasoning_effort`. The canonical values
+(`low`, `medium`, `high`) map to each provider’s native budget system.
+Thinking tokens stream as 🧠:
+
+``` python
+# Claude with thinking
+print("Claude: ", end='')
+r = await stream([user("What is 127 × 849?")], model='claude-sonnet-4-6', reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if p.type == PartType.thinking: print(f"\n🧠 {p.text[:150]}...")
+
+# Kimi with thinking — same interface
+print("\nKimi: ", end='')
+r = await stream([user("What is 127 × 849?")], model='accounts/fireworks/models/kimi-k2p5', vendor_name='fireworks_ai', reasoning_effort='low', max_tokens=8192)
+for p in r.message.content:
+    if p.type == PartType.thinking: print(f"\n🧠 {p.text[:150]}...")
+```
+
+    Claude: 🧠🧠🧠## Calculating 127 × 849
+
+    **Breaking it down:**
+    - 127 × 800 = 101,600
+    - 127 × 40 = 5,080
+    - 127 × 9 = 1,143
+
+    **Adding the parts:**
+    101,600 + 5,080 + 1,143 = **107,823**
+
+    🧠 127 × 849:
+    127 × 800 = 101,600
+    127 × 49 = 127 × 50 - 127 = 6,350 - 127 = 6,223
+    Total: 107,823...
+
+    Kimi: 🧠🧠🧠🧠🧠🧠🧠🧠🧠🧠$127 \times 849 = 107{,}823$
+
+    🧠 The user is asking for the product of 127 and 849. I need to calculate this.
+
+    Let me break it down:
+    127 × 849
+
+    I can use the distributive property:
+    12...
+
+## Web Search (Server Tools)
+
+OpenAI’s Responses API supports server-side web search. Server tool
+calls are normalized alongside regular tool calls:
+
+``` python
+ws_tools = [{"type": "web_search_preview"}]
+print("GPT + web search: ", end='')
+r = await stream([user("What is the latest Python release?")], model='gpt-4o-mini', tools=ws_tools, max_tokens=512)
+print(f"\nServer tools used: {[tc.name for tc in r.tool_calls if tc.server]}")
+```
+
+    GPT + web search: As of April 29, 2026, the latest stable release of Python is version 3.14.4, which was released on April 7, 2026. ([ivyml.space](https://ivyml.space/downloads/?utm_source=openai)) This version is currently in the "bugfix" phase, receiving full support for bug fixes and security updates. Python 3.14 was first released on October 7, 2025, and is scheduled to receive support until October 31, 2030. ([eol.wiki](https://eol.wiki/python/?utm_source=openai))
+
+    Python 3.15 is currently in the alpha development phase, with the first alpha release (3.15.0a1) planned for October 14, 2025. ([peps.python.org](https://peps.python.org/pep-0790/?utm_source=openai)) The stable release of Python 3.15 is expected in October 2026.
+
+    For more information on Python releases and their support statuses, you can visit the official Python documentation. ([python.org](https://www.python.org/doc/versions/?utm_source=openai)) 
+
+    Server tools used: ['web_search']
+
+## Caching (Anthropic)
+
+Anthropic supports prompt caching via `cache_control`. Pass it through
+`Part.data` — repeat calls with the same cached content save tokens:
+
+``` python
+# Cache a long system prompt (must be >1024 tokens for Anthropic caching)
+long_ctx = "You are an expert on the solar system. " * 200
+system = Part(type=PartType.text, text=long_ctx, data={'cache_control': {'type': 'ephemeral'}})
+
+print("Call 1: ", end='')
+r1 = await stream([user("What is Jupiter's mass?")], model='claude-sonnet-4-20250514', system=system, max_tokens=mtok)
+print(f"Usage: {r1.usage}")
+
+print("Call 2: ", end='')
+r2 = await stream([user("How about Saturn?")], model='claude-sonnet-4-20250514', system=system, max_tokens=mtok)
+print(f"Usage: {r2.usage}")
+```
+
+    Call 1: Jupiter's mass is approximately 1.898 × 10^27 kilograms (or 1,898,000,000,000,000,000,000,000,000 kg).
+
+    To put this in perspective:
+    - Jupiter is about 318 times more massive than Earth
+    - It contains more than twice the mass of all other planets in our solar system combined
+    - It's about 1/1047th the mass of our Sun
+
+    This enormous mass is what makes Jupiter such a dominant gravitational force in our solar system, allowing it to capture and hold onto its 95+ known moons and play a crucial role in shaping the orbits of other celestial bodies.
+    Usage: Usage(prompt_tokens=1814, completion_tokens=151, total_tokens=1965, cached_tokens=0, cache_creation_tokens=1802, reasoning_tokens=0, raw={'input_tokens': 12, 'cache_creation_input_tokens': 1802, 'cache_read_input_tokens': 0, 'output_tokens': 151})
+    Call 2: Saturn is truly one of the most spectacular planets in our solar system! Here are some key facts about this magnificent world:
+
+    **Basic characteristics:**
+    - The sixth planet from the Sun and second-largest in our solar system
+    - A gas giant composed primarily of hydrogen and helium
+    - About 9.5 times Earth's distance from the Sun
+    - Takes about 29.5 Earth years to orbit the Sun
+
+    **Famous ring system:**
+    - Saturn's rings are its most iconic feature - made of countless ice and rock particles
+    - The main rings span about 175,000 miles across but are surprisingly thin (often less than 30 feet thick)
+    - There are several distinct ring groups (A, B, C rings are the most prominent)
+    - The rings may be relatively young - possibly only 10-100 million years old
+
+    **Physical properties:**
+    - Diameter about 9 times larger than Earth
+    - Surprisingly low density - it would actually float in water!
+    - Rotates very quickly (about 10.5 hour days), causing it to bulge at the equator
+    - Has beautiful banded cloud patterns in its atmosphere
+
+    **Moons:**
+    - Saturn has 146 confirmed moons, including Titan (larger than Mercury, with lakes of liquid methane) and Enceladus (which shoots geysers of water ice from its south pole)
+
+    **Exploration:**
+    - Visited by Pioneer 11, Voyager 1 & 2, and most extensively by the Cassini mission (2004-2017)
+
+    What aspect of Saturn interests you most?
+    Usage: Usage(prompt_tokens=1812, completion_tokens=350, total_tokens=2162, cached_tokens=1802, cache_creation_tokens=0, reasoning_tokens=0, raw={'input_tokens': 10, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 1802, 'output_tokens': 350})
+
+## Media Inputs
+
+Send images to any provider that supports them. The canonical
+`input_image` part works everywhere:
+
+``` python
+img_url = "https://img.freepik.com/free-photo/mountain-range-body-water_53876-139760.jpg?semt=ais_hybrid&w=740&q=80"
+img_msg = Msg(role='user', content=[
+    Part(type=PartType.input_image, text=img_url),
+    Part(type=PartType.text, text="What do you see in this image?")
+])
+
+for name, kw in [('claude-sonnet-4-20250514', {}), ('gpt-4o-mini', {}), ('models/gemini-3-flash-preview', {})]:
+    print(f"{name:>30s}: ", end='')
+    r = await stream([img_msg], model=name, max_tokens=mtok, **kw)
+```
+
+          claude-sonnet-4-20250514: I see a beautiful, serene landscape featuring:
+
+    - A calm lake or body of water that creates perfect mirror-like reflections
+    - Snow-capped mountains in the background creating a dramatic backdrop
+    - Dense forest of evergreen trees (likely pine or fir) lining the shoreline
+    - A wooden deck or boardwalk in the foreground with weathered planks
+    - Clear blue sky with some clouds
+    - The scene has a peaceful, pristine quality typical of alpine or mountain lake environments
+
+    The composition creates a sense of tranquility and natural beauty, with the wooden platform providing a viewing point to take in this scenic mountain landscape. The perfect reflections in the still water double the visual impact of the mountains and trees.
+                       gpt-4o-mini: The image shows a serene landscape featuring a calm lake that reflects surrounding mountains and trees. The foreground includes a wooden deck or platform, which adds a sense of depth to the scene. In the background, the mountains are partially covered by mist, creating a peaceful and tranquil atmosphere. The overall color palette is dominated by blues and greens, evoking a natural and serene vibe.
+     models/gemini-3-flash-preview: This image is a serene landscape featuring a mountain lake. It is composed of several distinct layers:
+
+    *   **Foreground:** At the bottom of the image, there is a rustic **wooden deck or platform** made of dark brown, weathered planks. The perspective makes it look as though the viewer is standing on this deck looking out at the view.
+    *   **Middle Ground:** A **calm, blue lake** sits directly behind the deck. The water is so still that it acts like a mirror, perfectly reflecting the trees and mountains above it.
+    *   **Shoreline:** Along the edge of the water, there is a thin strip of golden-yellow grass or low-lying vegetation. This is followed by a **dense line of green trees**, including some tall, dark conifers.
+    *   **Background:** In the far distance, **massive mountains** rise up. They are a hazy blue-grey color, with their peaks covered in snow and ice. 
+    *   **Sky:** The sky is a very pale blue, almost white, with soft, bright light suggesting a sunny but perhaps slightly hazy day.
+
+    The overall atmosphere of the image is peaceful and majestic.
+
+`fastllm` supports four media part types via `PartType`. Provider
+support varies:
+
+| Part Type     | Anthropic | OpenAI Responses | OpenAI Chat | Gemini |
+|---------------|-----------|------------------|-------------|--------|
+| `input_image` | ✅        | ✅               | ✅          | ✅     |
+| `input_audio` | ❌        | ❌ (coming soon) | ✅          | ✅     |
+| `input_video` | ❌        | ❌               | ❌          | ✅     |
+| `input_file`  | ✅        | ✅               | ✅          | ✅     |
+
+All media parts accept either a URL or a base64 data URL in `Part.text`.
+Unsupported combinations raise a clear `ValueError`.