foil-sdk 0.5.0__tar.gz

foil_sdk-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Foil
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

foil_sdk-0.5.0/PKG-INFO

@@ -0,0 +1,665 @@
+Metadata-Version: 2.4
+Name: foil-sdk
+Version: 0.5.0
+Summary: Foil SDK for monitoring and logging AI model invocations
+Author: Foil
+License-Expression: MIT
+Project-URL: Homepage, https://getfoil.ai
+Project-URL: Documentation, https://docs.getfoil.ai
+Project-URL: Repository, https://github.com/getfoil/foil
+Keywords: ai,monitoring,logging,openai,llm,opentelemetry,tracing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.25.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Provides-Extra: otel
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "otel"
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "otel"
+Provides-Extra: openllmetry
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "openllmetry"
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "openllmetry"
+Requires-Dist: traceloop-sdk>=0.15.0; extra == "openllmetry"
+Dynamic: license-file
+
+# Foil Python SDK
+
+Python SDK for monitoring and logging AI agent invocations with Foil. Features distributed tracing, semantic search, custom evaluations, multimodal content support (images, documents), signals/feedback, and OpenAI integration.
+
+## Installation
+
+```bash
+pip install foil-sdk
+```
+
+Or install from source:
+
+```bash
+pip install -e /path/to/foil-sdk
+```
+
+## Quick Start
+
+```python
+from foil import Foil, create_foil_tracer, SpanKind
+
+# Create a tracer for your agent
+tracer = create_foil_tracer(
+    api_key="your-api-key",
+    agent_name="my-agent",
+    base_url="https://api.foil.dev/api",  # or your self-hosted URL
+)
+
+# Trace an agent execution
+async def my_agent():
+    async with tracer.trace("weather-query", input="User asked about weather") as ctx:
+        # Create an LLM span
+        span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="What is the weather?")
+
+        # ... call your LLM ...
+
+        await span.end(
+            output="The weather is sunny today!",
+            tokens={"prompt": 10, "completion": 8, "total": 18}
+        )
+
+        return "Agent completed"
+```
+
+## Table of Contents
+
+- [Distributed Tracing](#distributed-tracing)
+- [Semantic Search](#semantic-search)
+- [Custom Evaluations](#custom-evaluations)
+- [Multimodal Content](#multimodal-content)
+- [Signals & Feedback](#signals--feedback)
+- [OpenAI Integration](#openai-integration)
+- [Experiments (A/B Testing)](#experiments-ab-testing)
+- [Low-Level API](#low-level-api)
+- [API Reference](#api-reference)
+- [Examples](#examples)
+
+## Distributed Tracing
+
+### Using the Tracer
+
+```python
+from foil import create_foil_tracer, SpanKind
+
+tracer = create_foil_tracer(
+    api_key="your-api-key",
+    agent_name="customer-support-agent",
+)
+
+async with tracer.trace("support-query", input=user_message, session_id=conversation_id) as ctx:
+    # LLM span for the main model call
+    span = await ctx.start_span(SpanKind.LLM, "gpt-4-turbo", input=messages)
+
+    response = await openai.chat.completions.create(
+        model="gpt-4-turbo",
+        messages=messages,
+    )
+
+    await span.end(
+        output=response.choices[0].message.content,
+        tokens={
+            "prompt": response.usage.prompt_tokens,
+            "completion": response.usage.completion_tokens,
+            "total": response.usage.total_tokens,
+        }
+    )
+```
+
+### Span Kinds
+
+```python
+from foil import SpanKind
+
+SpanKind.AGENT      # Root agent span
+SpanKind.LLM        # Language model calls
+SpanKind.TOOL       # Tool/function executions
+SpanKind.CHAIN      # Chain of operations
+SpanKind.RETRIEVER  # RAG retrieval operations
+SpanKind.EMBEDDING  # Embedding model calls
+SpanKind.CUSTOM     # Custom operation types
+```
+
+### Nested Spans
+
+```python
+async with tracer.trace("order-lookup") as ctx:
+    # LLM decides to use a tool
+    llm_span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="Find order #12345")
+
+    # Tool execution (child of LLM span)
+    tool_span = await ctx.start_span(SpanKind.TOOL, "lookup_order", input={"order_id": "12345"})
+    result = await database.find_order("12345")
+    await tool_span.end(output=result)
+
+    await llm_span.end(output=f"Order found: {result}")
+```
+
+## Semantic Search
+
+Search through your traces using natural language queries. Foil automatically embeds your trace content and enables AI-powered semantic search.
+
+### Basic Search
+
+```python
+from foil import Foil
+
+foil = Foil(
+    api_key="your-api-key",
+    base_url="https://api.foil.dev/api",
+)
+
+# Search for conversations about a topic
+results = foil.semantic_search("conversations about refund requests")
+
+print(f"Found {len(results['results'])} matching traces")
+for result in results["results"]:
+    print(f"Trace: {result['traceId']}, Similarity: {result['similarity'] * 100:.1f}%")
+```
+
+### Search with Filters
+
+```python
+results = foil.semantic_search(
+    "user asking about pricing",
+    agent_id="customer-support-agent",  # Filter by specific agent
+    agent_name="support-bot",           # Or filter by agent name
+    from_date="2024-01-01",             # Start date (ISO format)
+    to_date="2024-12-31",               # End date (ISO format)
+    limit=10,                           # Max results (default: 20)
+    offset=0,                           # Pagination offset
+    threshold=0.4,                      # Min similarity score 0-1 (default: 0.3)
+)
+```
+
+### Find Similar Traces
+
+Find traces that are semantically similar to a specific trace:
+
+```python
+# Find traces similar to a known trace
+similar = foil.find_similar_traces(
+    "trace-abc-123",
+    limit=5,
+    threshold=0.4,
+)
+
+print(f"Found {len(similar['results'])} similar traces")
+for result in similar["results"]:
+    print(f"{result['traceId']}: {result['similarity'] * 100:.1f}% similar")
+```
+
+### Check Semantic Search Status
+
+```python
+# Get overall embedding status
+status = foil.get_semantic_search_status()
+print(f"Embedded: {status['embeddedSpans']} / {status['totalSpans']} spans")
+print(f"Coverage: {status['coveragePercent']}%")
+print(f"Ready: {status['ready']}")
+
+# Get status for specific agent
+agent_status = foil.get_semantic_search_status("my-agent-id")
+```
+
+## Custom Evaluations
+
+Define custom evaluation criteria to analyze your agent's responses. Evaluations run automatically on new traces and can be boolean, score-based, or categorical.
+
+### Get Evaluation Templates
+
+```python
+# List available pre-built evaluation templates
+templates = foil.get_evaluation_templates()
+
+for t in templates.get("templates", []):
+    print(f"{t['name']}: {t.get('description', 'N/A')}")
+```
+
+### Create a Boolean Evaluation
+
+```python
+evaluation = foil.create_evaluation("agent-123", {
+    "name": "professional_tone",
+    "description": "Checks if the response maintains a professional tone",
+    "prompt": """Evaluate the assistant's response for professional tone.
+
+Consider:
+- Is the language respectful and courteous?
+- Does it avoid slang or casual expressions?
+- Is it helpful without being condescending?
+
+Return true if professional, false otherwise.""",
+    "evaluationType": "boolean",
+    "enabled": True,
+})
+
+print(f"Created evaluation: {evaluation['evaluation']['name']}")
+```
+
+### Create a Score Evaluation (1-10)
+
+```python
+evaluation = foil.create_evaluation("agent-123", {
+    "name": "helpfulness_score",
+    "description": "Rates response helpfulness on a scale of 1-10",
+    "prompt": """Rate the helpfulness of the response on a scale of 1-10.
+
+Scoring:
+- 1-3: Unhelpful
+- 4-5: Somewhat helpful
+- 6-7: Helpful
+- 8-9: Very helpful
+- 10: Exceptional
+
+Return a single number from 1 to 10.""",
+    "evaluationType": "score",
+    "scoreMin": 1,
+    "scoreMax": 10,
+    "enabled": True,
+})
+```
+
+### Create a Category Evaluation
+
+```python
+evaluation = foil.create_evaluation("agent-123", {
+    "name": "response_intent",
+    "description": "Categorizes the type of response",
+    "prompt": """Classify the response into one category:
+
+- informational: Provides facts or explanations
+- action: Guides through a process
+- clarification: Asks for more information
+- escalation: Suggests human assistance
+- closure: Wraps up the conversation
+
+Return only the category name.""",
+    "evaluationType": "category",
+    "categories": ["informational", "action", "clarification", "escalation", "closure"],
+    "enabled": True,
+})
+```
+
+### Test an Evaluation
+
+Validate your evaluation prompt before enabling:
+
+```python
+result = foil.test_evaluation("agent-123", evaluation_id, {
+    "input": "How do I reset my password?",
+    "output": "I'd be happy to help! Go to Settings > Security > Reset Password.",
+})
+
+print(f"Result: {result['result']}")        # true/false, score, or category
+print(f"Reasoning: {result['reasoning']}")  # Explanation
+```
+
+### Add Few-Shot Examples
+
+Improve evaluation accuracy with examples:
+
+```python
+# Add a positive example
+foil.add_evaluation_example("agent-123", evaluation_id, {
+    "input": "Why is my order late?",
+    "output": "I apologize for the delay. Let me check the status for you.",
+    "expectedResult": True,
+    "reasoning": "Professional and helpful response",
+})
+
+# Add a negative example
+foil.add_evaluation_example("agent-123", evaluation_id, {
+    "input": "Why is my order late?",
+    "output": "idk check tracking yourself",
+    "expectedResult": False,
+    "reasoning": "Casual and unhelpful",
+})
+```
+
+### Manage Evaluations
+
+```python
+# List all evaluations for an agent
+evaluations = foil.get_agent_evaluations("agent-123")
+
+# Get specific evaluation details
+evaluation = foil.get_evaluation("agent-123", evaluation_id)
+
+# Update an evaluation
+foil.update_evaluation("agent-123", evaluation_id, {
+    "description": "Updated description",
+    "enabled": False,  # Disable temporarily
+})
+
+# Get evaluation analytics
+analytics = foil.get_evaluation_analytics("agent-123", evaluation_id)
+print(f"Total evaluations: {analytics.get('totalEvaluations', 0)}")
+print(f"Distribution: {analytics.get('distribution')}")
+
+# Clone a template
+cloned = foil.clone_evaluation_template("agent-123", template_id)
+
+# Remove an example
+foil.remove_evaluation_example("agent-123", evaluation_id, example_id)
+
+# Delete an evaluation
+foil.delete_evaluation("agent-123", evaluation_id)
+```
+
+## Multimodal Content
+
+Foil supports multimodal input/output including images, documents, and other media types.
+
+### Media Categories
+
+```python
+from foil import MediaCategory
+
+MediaCategory.IMAGE       # Images (png, jpg, gif, webp, etc.)
+MediaCategory.DOCUMENT    # Documents (pdf, doc, docx, etc.)
+MediaCategory.SPREADSHEET # Spreadsheets (xlsx, csv, etc.)
+MediaCategory.CODE        # Code files
+MediaCategory.AUDIO       # Audio files
+MediaCategory.VIDEO       # Video files
+MediaCategory.ARCHIVE     # Archives (zip, tar, etc.)
+MediaCategory.NOTEBOOK    # Jupyter notebooks
+MediaCategory.OTHER       # Other file types
+```
+
+### Uploading Media
+
+```python
+from foil import Foil
+
+foil = Foil(
+    api_key="your-api-key",
+    base_url="https://api.foil.dev/api",
+)
+
+# Upload from file path
+result = foil.upload_media("/path/to/image.png")
+print(result["mediaId"])   # 'media_abc123'
+print(result["category"])  # 'image'
+print(result["filename"])  # 'image.png'
+print(result["mimeType"])  # 'image/png'
+
+# Upload from bytes
+with open("/path/to/document.pdf", "rb") as f:
+    content = f.read()
+result = foil.upload_media(
+    content,
+    filename="document.pdf",
+    mime_type="application/pdf",
+)
+
+# Upload with trace/span association
+result = foil.upload_media(
+    "/path/to/image.png",
+    trace_id="trace_123",
+    span_id="span_456",
+    direction="input",  # or "output"
+)
+```
+
+### Content Blocks
+
+Use content blocks to create multimodal input/output:
+
+```python
+from foil import content, ContentBlock, MediaCategory
+
+# Create multimodal content array
+multimodal_input = content(
+    "Please analyze this image:",
+    ContentBlock.media(
+        upload_result["mediaId"],
+        category=MediaCategory.IMAGE,
+        filename="photo.jpg",
+        mime_type="image/jpeg",
+    ),
+    "Focus on the composition and colors."
+)
+```
+
+### Media Retrieval
+
+```python
+# Get media information
+media_info = foil.get_media(media_id)
+
+# Get presigned URL for download
+url_info = foil.get_media_url(media_id, "original")
+print(url_info["url"])  # Presigned S3 URL
+
+# Get extracted content URL (for documents)
+extracted_url = foil.get_media_url(media_id, "extracted")
+
+# Batch get multiple media
+batch_info = foil.batch_media_info([media_id1, media_id2, media_id3])
+```
+
+## Signals & Feedback
+
+Record user feedback and custom signals for your traces:
+
+```python
+from foil import Foil
+
+foil = Foil(api_key="your-api-key")
+
+# Record a signal for a specific trace
+foil.record_signal({
+    "traceId": "trace_123",
+    "signalName": "user_satisfaction",
+    "value": 4,
+    "signalType": "feedback",
+    "source": "user",
+})
+
+# Batch record signals
+foil.record_signal_batch([
+    {"traceId": "trace_123", "signalName": "thumbs", "value": True, "source": "user"},
+    {"traceId": "trace_123", "signalName": "rating", "value": 5, "source": "user"},
+])
+
+# Get signals for a trace
+signals = foil.get_trace_signals("trace_123")
+```
+
+## OpenAI Integration
+
+Automatic logging for OpenAI calls:
+
+```python
+from openai import OpenAI
+from foil import Foil
+
+foil = Foil(api_key="your-foil-api-key")
+client = foil.wrap_openai(OpenAI())
+
+# All chat completion calls are now automatically logged
+response = client.chat.completions.create(
+    model="gpt-4",
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+```
+
+Supports both streaming and non-streaming responses.
+
+## Experiments (A/B Testing)
+
+Get variant assignments for experiments:
+
+```python
+from foil import Foil
+
+foil = Foil(api_key="your-api-key")
+
+# Get assigned variant for a user
+assignment = foil.get_experiment_variant("experiment_123", user_id)
+
+if assignment["inExperiment"]:
+    print(assignment["variantName"])  # 'control' or 'treatment'
+    print(assignment["config"])       # {'model': 'gpt-4', 'temperature': 0.7}
+```
+
+## Low-Level API
+
+For more control, use the Foil client directly:
+
+```python
+from foil import Foil
+
+foil = Foil(
+    api_key="your-api-key",
+    base_url="https://api.foil.dev/api",
+)
+
+# Manual span management
+trace_id = foil.create_trace_id()
+span_id = foil.create_span_id()
+
+foil.start_span({
+    "spanId": span_id,
+    "traceId": trace_id,
+    "name": "gpt-4",
+    "agentName": "my-agent",
+    "spanKind": "llm",
+    "input": messages,
+})
+
+# ... do work ...
+
+foil.end_span({
+    "spanId": span_id,
+    "traceId": trace_id,
+    "agentName": "my-agent",
+    "output": response,
+    "tokens": {"prompt": 100, "completion": 50, "total": 150},
+})
+
+# Get trace data
+trace = foil.get_trace(trace_id)
+
+# List traces
+traces = foil.list_traces(
+    agent_name="my-agent",
+    limit=10,
+    from_date="2024-01-01T00:00:00Z",
+)
+```
+
+## API Reference
+
+### Foil Client
+
+#### Initialization
+
+```python
+foil = Foil(api_key="your-api-key", base_url="https://api.getfoil.ai/api")
+```
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `api_key` | str | Yes | Your Foil API key |
+| `base_url` | str | No | API base URL (default: https://api.getfoil.ai) |
+
+### Semantic Search Methods
+
+| Method | Description |
+|--------|-------------|
+| `semantic_search(query, **options)` | Search spans using natural language |
+| `find_similar_traces(trace_id, **options)` | Find traces similar to a given trace |
+| `get_semantic_search_status(agent_id=None)` | Get embedding statistics |
+
+### Custom Evaluation Methods
+
+| Method | Description |
+|--------|-------------|
+| `get_evaluation_templates()` | List available evaluation templates |
+| `get_agent_evaluations(agent_id)` | List evaluations for an agent |
+| `get_evaluation(agent_id, evaluation_id)` | Get evaluation details |
+| `create_evaluation(agent_id, data)` | Create a custom evaluation |
+| `update_evaluation(agent_id, evaluation_id, data)` | Update an evaluation |
+| `delete_evaluation(agent_id, evaluation_id)` | Delete an evaluation |
+| `test_evaluation(agent_id, evaluation_id, data)` | Test with sample data |
+| `clone_evaluation_template(agent_id, template_id)` | Clone a template |
+| `add_evaluation_example(agent_id, evaluation_id, data)` | Add few-shot example |
+| `remove_evaluation_example(agent_id, evaluation_id, example_id)` | Remove example |
+| `get_evaluation_analytics(agent_id, evaluation_id)` | Get evaluation metrics |
+
+### Media Methods
+
+| Method | Description |
+|--------|-------------|
+| `upload_media(file, **options)` | Upload media for multimodal content |
+| `get_media(media_id, content=None)` | Get media information |
+| `get_media_url(media_id, content="original")` | Get presigned download URL |
+| `batch_media_info(media_ids)` | Get info for multiple media |
+
+### Signal Methods
+
+| Method | Description |
+|--------|-------------|
+| `record_signal(data)` | Record a signal |
+| `record_signal_batch(signals)` | Record multiple signals |
+| `get_trace_signals(trace_id)` | Get signals for a trace |
+
+### Trace Methods
+
+| Method | Description |
+|--------|-------------|
+| `start_span(data)` | Start a span |
+| `end_span(data)` | End a span |
+| `get_trace(trace_id)` | Get trace with all spans |
+| `list_traces(**options)` | List traces with filters |
+
+### Experiment Methods
+
+| Method | Description |
+|--------|-------------|
+| `get_experiment_variant(experiment_id, identifier)` | Get variant assignment |
+
+### Legacy Methods (Deprecated)
+
+| Method | Replacement |
+|--------|-------------|
+| `start_invocation(data)` | Use `start_span(data)` |
+| `end_invocation(data)` | Use `end_span(data)` |
+
+## Examples
+
+See the `examples/` directory for complete working examples:
+
+- `semantic_search_example.py` - Semantic search usage
+- `custom_evaluations_example.py` - Custom evaluations
+
+Run examples:
+
+```bash
+export FOIL_API_KEY=your-api-key
+export FOIL_BASE_URL=https://api.getfoil.ai/api
+export FOIL_AGENT_ID=your-agent-id
+
+python examples/semantic_search_example.py
+python examples/custom_evaluations_example.py
+```
+
+## License
+
+MIT