connectonion 0.6.2__py3-none-any.whl → 0.6.3__py3-none-any.whl

connectonion/docs/debug/console.md

@@ -0,0 +1,129 @@
+# Console Output
+
+ConnectOnion shows you what's happening by default - just like FastAPI, npm, and cargo.
+
+## No Configuration Needed
+
+```python
+# Console output is always visible
+agent = Agent("assistant", tools=[my_tool])
+result = agent.input("Do something")
+```
+
+## What You See
+
+When you run an agent, you'll see the onion stack banner and execution flow:
+
+```
+  ○
+ ◎    my-assistant
+●     ────────────────────
+      connectonion v0.5.1
+      co/o4-mini · 3 tools
+      .co/logs/ · .co/evals/
+      ────────────────────
+
+[co] > "Generate a Python function"
+
+[co] ○ o4-mini                                    1/10
+[co] ● o4-mini · 1 tool · 150 tok · $0.0012       ⚡ 1.2s
+[co]   ▸ generate_code(language="python")         ✓ 0.12s
+
+[co] ═══════════════════════════════════════════════
+[co] ✓ complete · 150 tokens · $0.0012 · 2.3s
+```
+
+Console output helps you understand:
+- What the agent is doing
+- Which tools are being called
+- How long operations take
+- When errors occur
+
+## Enhanced Output with @xray
+
+Want more details? Use the `@xray` decorator:
+
+```python
+from connectonion import Agent, xray
+
+@xray
+def my_tool(query: str) -> str:
+    """This tool shows a Rich table with detailed info."""
+    return process(query)
+
+agent = Agent("assistant", tools=[my_tool])
+result = agent.input("Search for python")
+```
+
+Output includes a beautiful table:
+```
+╭──────────────────── @xray: my_tool ────────────────────╮
+│  agent       assistant                                 │
+│  task        Search for python                         │
+│  iteration   1                                         │
+│  ───────────────────────────────────────────────────   │
+│  query       python                                    │
+│  result      Found 10 results...                       │
+│  timing      340.2ms                                   │
+╰────────────────────────────────────────────────────────╯
+```
+
+## File Logging
+
+ConnectOnion automatically logs to three places:
+
+```python
+# Default: console + plain text + session YAML
+agent = Agent("assistant")
+
+# Custom log file path
+agent = Agent("assistant", log="agent.log")
+
+# Or use environment variable (highest priority)
+# CONNECTONION_LOG=agent.log python agent.py
+```
+
+Log locations:
+- Plain text: `.co/logs/{name}.log`
+- Session YAML: `.co/evals/{name}.yaml`
+
+## Quiet Mode
+
+For eval/testing, suppress console but keep session logging:
+
+```python
+# Quiet mode: no console, but sessions recorded for replay
+agent = Agent("assistant", quiet=True)
+```
+
+| quiet | log | Console | Plain Text | Sessions | Use Case |
+|-------|-----|---------|------------|----------|----------|
+| False | True/None | ✓ | ✓ | ✓ | Development (default) |
+| True | True/None | ✗ | ✗ | ✓ | Eval/testing |
+| False | False | ✓ | ✗ | ✗ | Benchmarking |
+
+## Why Console is On by Default
+
+**Design Philosophy:** Good UX means showing what's happening by default.
+
+When you run:
+- `npm install` - you see packages being installed
+- `cargo build` - you see compilation progress
+- `fastapi dev` - you see server logs
+
+Why should agents be silent? ConnectOnion follows the same principle - **visibility by default**.
+
+## What Changed?
+
+**Previous design (0.0.6 and earlier):**
+- Console was off by default
+- Required `debug=True` to see output
+- Confusing - output wasn't "debugging", it was normal operation
+
+**Current design (0.0.7+):**
+- Console is on by default (use `quiet=True` to suppress)
+- Shows what's happening (like FastAPI, npm, cargo)
+- `@xray` decorator adds enhanced Rich tables for specific tools
+- Session YAML logging for eval and replay
+
+**Rationale:** The console output isn't "debug" information - it's useful operation visibility that users expect. Hiding it by default created confusion and poor developer experience.

connectonion/docs/debug/eval-format.md

@@ -0,0 +1,178 @@
+# Eval YAML Format
+
+Evals are stored in `.co/evals/{input_slug}.yaml`. File is named from the first user input (slugified). Same input sequence = same file with run tracking. Log = Eval (same format, add `expected` field for tests).
+
+## File Structure
+
+```
+.co/evals/
+├── say_hello_to_alice.yaml              # Eval metadata + turns summary
+├── say_hello_to_alice/
+│   ├── run_1.yaml                       # Run 1: metadata + messages as JSON
+│   └── run_2.yaml                       # Run 2: metadata + messages as JSON
+├── open_hacker_news_and_summarize.yaml
+└── open_hacker_news_and_summarize/
+    └── run_1.yaml
+```
+
+## Eval Summary YAML
+
+```yaml
+name: say_hello_to_alice
+created: '2025-12-25 11:56:26'
+updated: '2025-12-25 11:56:56'
+runs: 2
+model: gemini-2.5-flash
+turns:
+- input: Say hello to Alice
+  run: 2
+  output: Hello, Alice!
+  tools_called:
+  - greet(name='Alice')
+  expected: ''                            # For eval: expected output
+  evaluation: ''                          # For eval: pass/fail/notes
+  meta: '{"tokens": 155, "cost": 0.0, "duration_ms": 3595, "ts": "2025-12-25 11:56:56"}'
+  history:
+  - run: 1
+    output: Hello, Alice!
+    tools_called:
+    - greet(name='Alice')
+    expected: ''
+    evaluation: ''
+    meta: '{"tokens": 155, "cost": 0.0, "duration_ms": 4438, "ts": "2025-12-25 11:56:26"}'
+```
+
+The `meta` field is a compact JSON string containing tokens, cost, duration, and timestamp.
+
+## Run YAML Format
+
+Each run has its own YAML file with metadata and messages:
+
+```yaml
+system_prompt: "You are a helpful assistant..."
+model: "claude-sonnet-4-5"
+cwd: "/path/to/project"
+tokens: 1314
+cost: 0.0015
+duration_ms: 6162
+timestamp: "2025-12-25 12:50:44"
+messages: |
+  [
+    {"role": "system", "content": "You are a helpful assistant..."},
+    {"role": "user", "content": "Say hello to Alice"},
+    {"role": "assistant", "tool_calls": [{"id": "...", "function": {"name": "greet", "arguments": "{\"name\": \"Alice\"}"}}]},
+    {"role": "tool", "content": "Hello, Alice!", "tool_call_id": "..."}
+  ]
+```
+
+**Key fields:**
+- `system_prompt`: The agent's system prompt (easy to access)
+- `model`: Which model was used
+- `cwd`: Working directory (for re-execution)
+- `tokens`, `cost`, `duration_ms`: Performance metrics
+- `messages`: Full message history as multi-line JSON (API format)
+
+## Design Rationale
+
+### Why file naming from input (not agent name)?
+
+Same input = same eval file. This enables:
+- **Run comparison**: See how outputs change across runs
+- **Regression testing**: Same input should produce consistent results
+- **History tracking**: Previous runs preserved in `history` array
+
+### Why YAML with embedded JSON for messages?
+
+1. **Clean separation**: YAML for readable metadata, JSON for API-compatible messages
+2. **Easy to navigate**: Multi-line JSON is scannable
+3. **Reproducibility**: `cwd` and `system_prompt` stored for re-execution
+4. **Single file per run**: No need to manage multiple files
+
+### Why runs and history?
+
+```yaml
+turns:
+- input: Say hello to Alice
+  run: 2                    # Latest run is the main entry
+  output: Hello, Alice!
+  history:                  # Previous runs for comparison
+  - run: 1
+    output: Hello, Alice!
+```
+
+- Latest run is always at top level (easy to see current state)
+- History preserves all previous runs for regression analysis
+- Compare outputs across model versions or code changes
+
+## Usage
+
+### Quick check latest run
+```bash
+head -30 .co/evals/say_hello_to_alice.yaml
+```
+
+### Load messages for replay
+```python
+import json
+import yaml
+
+def load_messages(eval_dir: str, run: int = 1) -> list:
+    """Load messages from run YAML file."""
+    with open(f"{eval_dir}/run_{run}.yaml") as f:
+        data = yaml.safe_load(f)
+    return json.loads(data['messages'])
+
+# Load run 1 messages
+messages = load_messages(".co/evals/say_hello_to_alice", run=1)
+```
+
+### Load run metadata
+```python
+import yaml
+
+with open('.co/evals/say_hello_to_alice/run_1.yaml') as f:
+    run_data = yaml.safe_load(f)
+
+print(f"Model: {run_data['model']}")
+print(f"System prompt: {run_data['system_prompt']}")
+print(f"CWD: {run_data['cwd']}")
+print(f"Tokens: {run_data['tokens']}, Cost: ${run_data['cost']}")
+```
+
+### Compare runs
+```python
+import yaml
+
+with open('.co/evals/say_hello_to_alice.yaml') as f:
+    data = yaml.safe_load(f)
+
+turn = data['turns'][0]
+print(f"Run {turn['run']}: {turn['output']}")
+
+for hist in turn.get('history', []):
+    print(f"Run {hist['run']}: {hist['output']}")
+```
+
+### Add expected results for eval
+```yaml
+turns:
+- input: Say hello to Alice
+  run: 2
+  output: Hello, Alice!
+  expected: "Hello, Alice!"           # What we expect
+  evaluation: "pass"                   # Did it pass?
+```
+
+### Parse metadata
+```python
+import json
+
+turn = data['turns'][0]
+meta = json.loads(turn['meta'])
+print(f"Tokens: {meta['tokens']}, Cost: ${meta['cost']}, Duration: {meta['duration_ms']}ms")
+```
+
+### Get total runs
+```bash
+yq '.runs' .co/evals/say_hello_to_alice.yaml
+```

connectonion/docs/debug/eval.md

@@ -0,0 +1,230 @@
+# Evals
+
+Run and manage agent evals with `co eval`. Test agent behavior, track results over time, use LLM-as-judge for semantic evaluation.
+
+## Quick Start
+
+```bash
+# Run all evals in .co/evals/
+co eval
+
+# Run specific eval
+co eval wikipedia_lookup
+
+# Run with specific agent file
+co eval --agent my_agent.py
+```
+
+## Creating Evals
+
+Create YAML files in `.co/evals/`:
+
+```yaml
+# .co/evals/wikipedia_lookup.yaml
+name: wikipedia_lookup
+agent: agent.py
+turns:
+- input: Go to Wikipedia and find when the Eiffel Tower was built.
+  expected: Contains 1889 or late 1880s
+```
+
+### Minimal Eval
+
+```yaml
+name: simple_greeting
+agent: agent.py
+turns:
+- input: Say hello to Alice
+  expected: Greeting includes Alice's name
+```
+
+### Multi-Turn Conversation
+
+Turns run sequentially within the same agent session, simulating a conversation:
+
+```yaml
+name: multi_turn_chat
+agent: agent.py
+turns:
+- input: My name is Bob
+  expected: Acknowledges the name Bob
+- input: What's my name?
+  expected: Remembers and says Bob
+```
+
+## YAML Format
+
+```yaml
+name: eval_name                    # Eval identifier
+agent: agent.py                    # Path to agent file (required)
+created: '2025-12-25 14:30:00'     # Auto-generated
+updated: '2025-12-25 18:35:58'     # Auto-updated on each run
+runs: 8                            # Total run count
+
+turns:
+- input: User input text           # What to send to agent
+  output: Agent response           # Captured from agent
+  expected: What we expect         # For LLM judge evaluation
+  pass: true                       # Judge result (true/false)
+  analysis: Output contains...     # Judge explanation
+  tools_called:                    # Tools the agent used
+  - open_browser(headless=False)
+  - go_to(url='https://...')
+  tokens: 62533                    # Total tokens used
+  cost: 0.0491                     # Total cost in USD
+  ts: '2025-12-25 18:35:58'        # Timestamp
+  run: 8                           # Run number
+  history: '[...]'                 # Previous runs (JSON array)
+```
+
+## LLM-as-Judge
+
+When `expected` is set, an LLM judges if the output satisfies the criteria:
+
+```yaml
+turns:
+- input: What's 2+2?
+  expected: Answer is 4
+```
+
+The judge considers:
+- Semantic similarity (not exact match)
+- Key information presence
+- Intent fulfillment
+
+Results stored as:
+- `pass: true/false` - Did it pass?
+- `analysis: "..."` - Why it passed/failed
+
+## History Tracking
+
+Each run is preserved in `history` as a JSON array:
+
+```yaml
+history: |
+  [
+  {"ts": "2025-12-25 18:35:58", "pass": true, "tokens": 62584, "cost": 0.05},
+  {"ts": "2025-12-25 18:25:06", "pass": true, "tokens": 54879, "cost": 0.04},
+  {"ts": "2025-12-25 18:20:25", "pass": false, "tokens": 40147, "cost": 0.03}]
+```
+
+Use history to:
+- Track stability across runs
+- Compare token usage and costs
+- Identify regressions
+
+## Agent File Structure
+
+Your agent file must export an `agent` variable:
+
+```python
+# agent.py
+from connectonion import Agent
+
+agent = Agent(
+    name="my_agent",
+    tools=[...],
+    model="co/gemini-2.5-pro"
+)
+
+if __name__ == "__main__":
+    agent.input("Interactive mode...")
+```
+
+## CLI Options
+
+```bash
+# Run all evals
+co eval
+
+# Run specific eval by name
+co eval my_eval_name
+
+# Override agent file (ignores YAML agent field)
+co eval --agent custom_agent.py
+co eval -a custom_agent.py
+```
+
+## Output
+
+```
+Loading: agent.py
+Running: wikipedia_lookup
+  input: Go to Wikipedia and find when the Eiffel Tower...
+  ✓ Output contains 1889 and height information...
+✓ wikipedia_lookup completed
+
+                    Eval Results
+┌──────────────────┬────────┬────────────┬────────────┐
+│ Eval             │ Status │ Expected   │ Output     │
+├──────────────────┼────────┼────────────┼────────────┤
+│ wikipedia_lookup │ ✓ pass │ Contains...│ The Eiff...│
+└──────────────────┴────────┴────────────┴────────────┘
+
+✓ 1 passed
+```
+
+## Best Practices
+
+### Write Good Expected Criteria
+
+```yaml
+# Bad - too specific
+expected: "The Eiffel Tower was built in 1889"
+
+# Good - semantic criteria
+expected: Contains the year 1889 and mentions Paris or France
+```
+
+### Keep Evals Focused
+
+```yaml
+# Bad - testing too many things
+- input: Go to Wikipedia, find the Eiffel Tower, take a screenshot, and summarize
+
+# Good - one thing at a time
+- input: Go to Wikipedia and find the Eiffel Tower height
+  expected: Height around 300-330 meters
+```
+
+### Use Multi-Turn for Conversations
+
+```yaml
+turns:
+- input: Remember that my favorite color is blue
+  expected: Acknowledges the color preference
+- input: What's my favorite color?
+  expected: Says blue
+```
+
+## File Organization
+
+```
+.co/evals/
+├── 1_basic_greeting.yaml      # Number prefix for ordering
+├── 2_wikipedia_lookup.yaml
+├── 3_multi_turn_chat.yaml
+└── 4_complex_workflow.yaml
+```
+
+## Troubleshooting
+
+### "No agent specified"
+
+Add `agent: agent.py` to YAML or use `--agent` flag:
+
+```bash
+co eval --agent my_agent.py
+```
+
+### "No 'agent' instance found"
+
+Ensure your agent file exports `agent`:
+
+```python
+agent = Agent("name", ...)  # Must be named 'agent'
+```
+
+### Eval passes but shouldn't (or vice versa)
+
+Check the `analysis` field to understand the judge's reasoning. Adjust `expected` to be more specific.