synth-ai 0.2.16__tar.gz → 0.2.19__tar.gz

synth_ai-0.2.19/PKG-INFO

@@ -0,0 +1,296 @@
+Metadata-Version: 2.4
+Name: synth-ai
+Version: 0.2.19
+Summary: RL as a service SDK - Core AI functionality and tracing
+Author-email: Synth AI <josh@usesynth.ai>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/synth-laboratories/synth-ai
+Project-URL: Repository, https://github.com/synth-laboratories/synth-ai
+Project-URL: Issues, https://github.com/synth-laboratories/synth-ai/issues
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: requests>=2.32.3
+Requires-Dist: urllib3>=2.3.0
+Requires-Dist: certifi>=2024.8.30
+Requires-Dist: tqdm>=4.66.4
+Requires-Dist: jsonschema>=4.23.0
+Requires-Dist: backoff>=2.0.0
+Requires-Dist: typing_extensions>=4.0.0
+Requires-Dist: rich>=13.9.0
+Requires-Dist: openai>=1.99.0
+Requires-Dist: anthropic>=0.42.0
+Requires-Dist: langfuse<3.0.0,>=2.53.9
+Requires-Dist: opentelemetry-api>=1.26.0
+Requires-Dist: opentelemetry-sdk>=1.26.0
+Requires-Dist: diskcache>=5.6.3
+Requires-Dist: groq>=0.30.0
+Requires-Dist: google-genai>=1.26.0
+Requires-Dist: together>=1.5.21
+Requires-Dist: mistralai>=1.9.2
+Requires-Dist: fastapi>=0.115.12
+Requires-Dist: uvicorn>=0.34.2
+Requires-Dist: numpy>=2.2.3
+Requires-Dist: networkx>=3.4.2
+Requires-Dist: redis>=6.2.0
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: ty>=0.0.1a5
+Requires-Dist: toml>=0.10.2
+Requires-Dist: sqlalchemy>=2.0.42
+Requires-Dist: aiosqlite>=0.21.0
+Requires-Dist: greenlet>=3.2.3
+Requires-Dist: libsql>=0.1.8
+Requires-Dist: pynacl>=1.5.0
+Requires-Dist: google-api-core>=2.25.1
+Requires-Dist: google-generativeai>=0.8.5
+Requires-Dist: crafter>=1.8.3
+Requires-Dist: click<8.2,>=8.1.7
+Requires-Dist: openai-harmony>=0.0.1
+Requires-Dist: asyncpg>=0.30.0
+Requires-Dist: aiohttp>=3.8.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: datasets>=4.0.0
+Requires-Dist: transformers>=4.56.1
+Requires-Dist: modal<2.0.0,>=1.1.4
+Requires-Dist: pyboy>=2.6.0
+Requires-Dist: setuptools>=80.9.0
+Requires-Dist: libsql-experimental>=0.0.55
+Provides-Extra: dev
+Requires-Dist: build>=1.2.2.post1; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Requires-Dist: keyring>=24.0.0; extra == "dev"
+Requires-Dist: pytest>=8.3.3; extra == "dev"
+Requires-Dist: pytest-xdist>=3.6.1; extra == "dev"
+Requires-Dist: pytest-timeout>=2.3.1; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: pyright>=1.1.350; extra == "dev"
+Requires-Dist: coverage[toml]>=7.3.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Provides-Extra: research
+Requires-Dist: crafter>=1.8.3; extra == "research"
+Requires-Dist: datasets>=4.0.0; extra == "research"
+Provides-Extra: swe
+Requires-Dist: morphcloud>=0.1.3; extra == "swe"
+Requires-Dist: swebench>=2.3.0; extra == "swe"
+Provides-Extra: all
+Requires-Dist: crafter>=1.8.3; extra == "all"
+Requires-Dist: datasets>=4.0.0; extra == "all"
+Requires-Dist: morphcloud>=0.1.3; extra == "all"
+Requires-Dist: swebench>=2.3.0; extra == "all"
+Provides-Extra: analytics
+Requires-Dist: pandas>=2.2.3; extra == "analytics"
+Dynamic: license-file
+
+# Synth-AI SDK
+
+[![Python](https://img.shields.io/badge/python-3.11+-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![PyPI](https://img.shields.io/badge/PyPI-0.2.10-orange)](https://pypi.org/project/synth-ai/)
+![Coverage](https://img.shields.io/badge/coverage-9.09%25-red)
+![Tests](https://img.shields.io/badge/tests-37%2F38%20passing-brightgreen)
+![Blacksmith CI](https://img.shields.io/badge/CI-Blacksmith%20Worker-blue)
+
+> **Synth-AI** — Reinforcement Learning-as-a-Service for agents.  
+> **Docs:** [Get Started →](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+## 🚀 Install version 0.2.16
+
+```bash
+pip install synth-ai
+# or
+uv add synth-ai
+```
+
+**Import:**
+
+```python
+import synth_ai
+```
+
+**CLI (with uvx):**
+
+```bash
+uvx synth-ai setup
+uvx synth-ai demo
+uvx synth-ai deploy
+uvx synth-ai run
+uvx synth-ai baseline  # For coding agents: get baseline scores
+```
+
+> Full quickstart: [https://docs.usesynth.ai/sdk/get-started](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+When you run `uvx synth-ai setup`, the SDK opens your browser to the Synth dashboard for a one‑time pairing (handshake) with your signed‑in session. The SDK will automatically:
+
+
+Fast and effective reinforcement learning for agents, via an API.  
+Easily scale GPU topologies, train multi-node, and integrate with existing agent software.
+
+### Highlights
+
+- Scale GPU topologies (A10Gs, H100s, multi-node available on request)
+- Thin FastAPI wrapper integration
+- Supports OSS models like **Qwen3** (GPT-OSS GA soon)
+- Own your trained models
+
+---
+
+## ⚙️ Getting Started
+
+Synth-AI ships with a built-in RL example: training **Qwen3-0.6B** on math reasoning.
+
+1. Create accounts at [Synth](https://usesynth.ai) and [Modal](https://modal.com)
+2. Then run:
+
+   ```bash
+   uvx synth-ai demo
+   uvx synth-ai setup
+   uvx synth-ai deploy
+   uvx synth-ai run
+   ```
+
+3. To walk through your first RL run, see  
+   👉 [Synth-AI SDK Docs](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+## 🤖 For Coding Agents: Get Started with Baselines
+
+**Baselines** are the fastest way for coding agents to evaluate changes and measure improvement on Synth tasks.
+
+### Why Use Baselines?
+
+Baselines provide a **self-contained evaluation system** that:
+- ✅ **No infrastructure required** — runs locally, no deployed task app needed
+- ✅ **Quick feedback loop** — get task-by-task results in seconds
+- ✅ **Compare changes** — establish a baseline score before making modifications
+- ✅ **Auto-discoverable** — finds baseline files automatically in your codebase
+
+### Quick Start for Coding Agents
+
+```bash
+# 1. List available baselines
+uvx synth-ai baseline list
+
+# 2. Run a quick 3-task baseline to get started
+uvx synth-ai baseline banking77 --split train --seeds 0,1,2
+
+# 3. Get your baseline score (full train split)
+uvx synth-ai baseline banking77 --split train
+
+# 4. Make your changes to the code...
+
+# 5. Re-run to compare performance
+uvx synth-ai baseline banking77 --split train --output results_after.json
+```
+
+### Available Baselines
+
+```bash
+# Filter by task type
+uvx synth-ai baseline list --tag rl          # RL tasks
+uvx synth-ai baseline list --tag nlp         # NLP tasks
+uvx synth-ai baseline list --tag vision      # Vision tasks
+
+# Run specific baselines
+uvx synth-ai baseline warming_up_to_rl       # Crafter survival game
+uvx synth-ai baseline pokemon_vl             # Pokemon Red (vision)
+uvx synth-ai baseline gepa                   # Banking77 classification
+```
+
+### Baseline Results
+
+Each baseline run provides:
+- **Task-by-task results** — see exactly which seeds succeed/fail
+- **Aggregate metrics** — success rate, mean/std rewards, total tasks
+- **Serializable output** — save to JSON with `--output results.json`
+- **Model comparison** — test different models with `--model`
+
+Example output:
+```
+============================================================
+Baseline Evaluation: Banking77 Intent Classification
+============================================================
+Split(s): train
+Tasks: 10
+Success: 8/10
+Execution time: 12.34s
+
+Aggregate Metrics:
+  mean_outcome_reward: 0.8000
+  success_rate: 0.8000
+  total_tasks: 10
+```
+
+### Creating Custom Baselines
+
+Coding agents can create new baseline files to test custom tasks:
+
+```python
+# my_task_baseline.py
+from synth_ai.baseline import BaselineConfig, BaselineTaskRunner, DataSplit, TaskResult
+
+class MyTaskRunner(BaselineTaskRunner):
+    async def run_task(self, seed: int) -> TaskResult:
+        # Your task logic here
+        return TaskResult(...)
+
+my_baseline = BaselineConfig(
+    baseline_id="my_task",
+    name="My Custom Task",
+    description="Evaluate my custom task",
+    task_runner=MyTaskRunner,
+    splits={
+        "train": DataSplit(name="train", seeds=list(range(10))),
+    },
+)
+```
+
+Place this file in `examples/baseline/` or name it `*_baseline.py` for auto-discovery.
+
+---
+
+## 🔐 SDK → Dashboard Pairing
+
+When you run `uvx synth-ai setup` (or legacy `uvx synth-ai rl_demo setup`):
+
+- The SDK opens your browser to the Synth dashboard to pair your SDK with your signed-in session.
+- Automatically detects your **user + organization**
+- Ensures both **API keys** exist
+- Writes them to your project’s `.env` as:
+
+  ```
+  SYNTH_API_KEY=
+  ENVIRONMENT_API_KEY=
+  ```
+
+✅ No keys printed or requested interactively — all handled via browser pairing.
+
+### Environment overrides
+
+- `SYNTH_CANONICAL_ORIGIN` → override dashboard base URL (default: https://www.usesynth.ai/dashboard)
+- `SYNTH_CANONICAL_DEV` → `1|true|on` to use local dashboard (http://localhost:3000)
+
+---
+
+## 📚 Documentation
+
+- **SDK Docs:** [https://docs.usesynth.ai/sdk/get-started](https://docs.usesynth.ai/sdk/get-started)
+- **CLI Reference:** [https://docs.usesynth.ai/cli](https://docs.usesynth.ai/cli)
+- **API Reference:** [https://docs.usesynth.ai/api](https://docs.usesynth.ai/api)
+- **Changelog:** [https://docs.usesynth.ai/changelog](https://docs.usesynth.ai/changelog)
+
+---
+
+## 🧠 Meta
+
+- Package: [`synth-ai`](https://pypi.org/project/synth-ai)
+- Import: `synth_ai`
+- Source: [github.com/synth-laboratories/synth-ai](https://github.com/synth-laboratories/synth-ai)
+- License: MIT

synth_ai-0.2.19/README.md

@@ -0,0 +1,210 @@
+# Synth-AI SDK
+
+[![Python](https://img.shields.io/badge/python-3.11+-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![PyPI](https://img.shields.io/badge/PyPI-0.2.10-orange)](https://pypi.org/project/synth-ai/)
+![Coverage](https://img.shields.io/badge/coverage-9.09%25-red)
+![Tests](https://img.shields.io/badge/tests-37%2F38%20passing-brightgreen)
+![Blacksmith CI](https://img.shields.io/badge/CI-Blacksmith%20Worker-blue)
+
+> **Synth-AI** — Reinforcement Learning-as-a-Service for agents.  
+> **Docs:** [Get Started →](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+## 🚀 Install version 0.2.16
+
+```bash
+pip install synth-ai
+# or
+uv add synth-ai
+```
+
+**Import:**
+
+```python
+import synth_ai
+```
+
+**CLI (with uvx):**
+
+```bash
+uvx synth-ai setup
+uvx synth-ai demo
+uvx synth-ai deploy
+uvx synth-ai run
+uvx synth-ai baseline  # For coding agents: get baseline scores
+```
+
+> Full quickstart: [https://docs.usesynth.ai/sdk/get-started](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+When you run `uvx synth-ai setup`, the SDK opens your browser to the Synth dashboard for a one‑time pairing (handshake) with your signed‑in session. The SDK will automatically:
+
+
+Fast and effective reinforcement learning for agents, via an API.  
+Easily scale GPU topologies, train multi-node, and integrate with existing agent software.
+
+### Highlights
+
+- Scale GPU topologies (A10Gs, H100s, multi-node available on request)
+- Thin FastAPI wrapper integration
+- Supports OSS models like **Qwen3** (GPT-OSS GA soon)
+- Own your trained models
+
+---
+
+## ⚙️ Getting Started
+
+Synth-AI ships with a built-in RL example: training **Qwen3-0.6B** on math reasoning.
+
+1. Create accounts at [Synth](https://usesynth.ai) and [Modal](https://modal.com)
+2. Then run:
+
+   ```bash
+   uvx synth-ai demo
+   uvx synth-ai setup
+   uvx synth-ai deploy
+   uvx synth-ai run
+   ```
+
+3. To walk through your first RL run, see  
+   👉 [Synth-AI SDK Docs](https://docs.usesynth.ai/sdk/get-started)
+
+---
+
+## 🤖 For Coding Agents: Get Started with Baselines
+
+**Baselines** are the fastest way for coding agents to evaluate changes and measure improvement on Synth tasks.
+
+### Why Use Baselines?
+
+Baselines provide a **self-contained evaluation system** that:
+- ✅ **No infrastructure required** — runs locally, no deployed task app needed
+- ✅ **Quick feedback loop** — get task-by-task results in seconds
+- ✅ **Compare changes** — establish a baseline score before making modifications
+- ✅ **Auto-discoverable** — finds baseline files automatically in your codebase
+
+### Quick Start for Coding Agents
+
+```bash
+# 1. List available baselines
+uvx synth-ai baseline list
+
+# 2. Run a quick 3-task baseline to get started
+uvx synth-ai baseline banking77 --split train --seeds 0,1,2
+
+# 3. Get your baseline score (full train split)
+uvx synth-ai baseline banking77 --split train
+
+# 4. Make your changes to the code...
+
+# 5. Re-run to compare performance
+uvx synth-ai baseline banking77 --split train --output results_after.json
+```
+
+### Available Baselines
+
+```bash
+# Filter by task type
+uvx synth-ai baseline list --tag rl          # RL tasks
+uvx synth-ai baseline list --tag nlp         # NLP tasks
+uvx synth-ai baseline list --tag vision      # Vision tasks
+
+# Run specific baselines
+uvx synth-ai baseline warming_up_to_rl       # Crafter survival game
+uvx synth-ai baseline pokemon_vl             # Pokemon Red (vision)
+uvx synth-ai baseline gepa                   # Banking77 classification
+```
+
+### Baseline Results
+
+Each baseline run provides:
+- **Task-by-task results** — see exactly which seeds succeed/fail
+- **Aggregate metrics** — success rate, mean/std rewards, total tasks
+- **Serializable output** — save to JSON with `--output results.json`
+- **Model comparison** — test different models with `--model`
+
+Example output:
+```
+============================================================
+Baseline Evaluation: Banking77 Intent Classification
+============================================================
+Split(s): train
+Tasks: 10
+Success: 8/10
+Execution time: 12.34s
+
+Aggregate Metrics:
+  mean_outcome_reward: 0.8000
+  success_rate: 0.8000
+  total_tasks: 10
+```
+
+### Creating Custom Baselines
+
+Coding agents can create new baseline files to test custom tasks:
+
+```python
+# my_task_baseline.py
+from synth_ai.baseline import BaselineConfig, BaselineTaskRunner, DataSplit, TaskResult
+
+class MyTaskRunner(BaselineTaskRunner):
+    async def run_task(self, seed: int) -> TaskResult:
+        # Your task logic here
+        return TaskResult(...)
+
+my_baseline = BaselineConfig(
+    baseline_id="my_task",
+    name="My Custom Task",
+    description="Evaluate my custom task",
+    task_runner=MyTaskRunner,
+    splits={
+        "train": DataSplit(name="train", seeds=list(range(10))),
+    },
+)
+```
+
+Place this file in `examples/baseline/` or name it `*_baseline.py` for auto-discovery.
+
+---
+
+## 🔐 SDK → Dashboard Pairing
+
+When you run `uvx synth-ai setup` (or legacy `uvx synth-ai rl_demo setup`):
+
+- The SDK opens your browser to the Synth dashboard to pair your SDK with your signed-in session.
+- Automatically detects your **user + organization**
+- Ensures both **API keys** exist
+- Writes them to your project’s `.env` as:
+
+  ```
+  SYNTH_API_KEY=
+  ENVIRONMENT_API_KEY=
+  ```
+
+✅ No keys printed or requested interactively — all handled via browser pairing.
+
+### Environment overrides
+
+- `SYNTH_CANONICAL_ORIGIN` → override dashboard base URL (default: https://www.usesynth.ai/dashboard)
+- `SYNTH_CANONICAL_DEV` → `1|true|on` to use local dashboard (http://localhost:3000)
+
+---
+
+## 📚 Documentation
+
+- **SDK Docs:** [https://docs.usesynth.ai/sdk/get-started](https://docs.usesynth.ai/sdk/get-started)
+- **CLI Reference:** [https://docs.usesynth.ai/cli](https://docs.usesynth.ai/cli)
+- **API Reference:** [https://docs.usesynth.ai/api](https://docs.usesynth.ai/api)
+- **Changelog:** [https://docs.usesynth.ai/changelog](https://docs.usesynth.ai/changelog)
+
+---
+
+## 🧠 Meta
+
+- Package: [`synth-ai`](https://pypi.org/project/synth-ai)
+- Import: `synth_ai`
+- Source: [github.com/synth-laboratories/synth-ai](https://github.com/synth-laboratories/synth-ai)
+- License: MIT

synth_ai-0.2.19/examples/analyze_semantic_words.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+# Run Crafter agent and analyze semantic map word distribution
+# This script demonstrates semantic analysis of agent observations
+# Output: Markdown tables and JSON data (no plotting dependencies)
+
+echo "🔍 Analyzing semantic map words from Crafter agent..."
+echo "Make sure the synth-ai service is running: uvx synth-ai deploy --runtime uvicorn"
+echo ""
+
+cd synth_ai/environments/examples/crafter_classic/agent_demos/
+
+# Run the semantic analysis (markdown output only)
+python analyze_semantic_words_markdown.py --model gemini-1.5-flash --episodes 3 --max-turns 30
+
+echo ""
+echo "✅ Analysis complete! Check the generated markdown report and JSON files."