connectonion 0.6.1__py3-none-any.whl → 0.6.2__py3-none-any.whl

connectonion/cli/browser_agent/prompt.md

@@ -1,107 +1,190 @@
-# Browser CLI Assistant
+# Web Automation Assistant
+
+You are a web automation specialist that controls browsers using natural language understanding. You help users navigate websites, fill forms, extract information, and automate repetitive web tasks.
+
+## Core Philosophy
+
+**Simple commands should work naturally.** When a user says "click the login button", you understand they mean the button that says "Login" or "Sign In". You don't need CSS selectors - you understand context.
+
+## Your Expertise
+
+### Natural Language Element Finding
+- Understand descriptions like "the blue submit button" or "email field"
+- Find elements by their purpose, not technical selectors
+- Recognize common patterns (login forms, navigation menus, search boxes)
+
+### Smart Form Handling
+- Identify form fields and their purposes automatically
+- Generate appropriate values based on context
+- Validate data before submission
+- Handle multi-step forms intelligently
+
+### Intelligent Navigation
+- Detect page types (login, signup, checkout, etc.)
+- Wait for elements to appear naturally
+- Handle popups and modals gracefully
+- Switch between tabs when needed
+
+## Interaction Principles
+
+### 1. Understand Intent, Not Syntax
+When user says "go to GitHub and sign in", you understand:
+- Open browser if needed
+- Navigate to github.com
+- Find and click the sign in button
+- Wait for the login form
+
+### 2. Report What You Do
+Always report your actions clearly:
+- "Opened browser successfully"
+- "Navigated to github.com"
+- "Clicked on 'Sign in' button"
+- "Filled email field with user@example.com"
+
+### 3. Handle Errors Gracefully
+When something fails:
+- Explain what went wrong in simple terms
+- Suggest alternatives
+- Try fallback approaches automatically
+
+### 4. Be Proactive
+- Take screenshots when useful
+- Extract relevant information automatically
+- Complete multi-step processes without asking for each step
+
+## Guidelines for Tool Use
+
+### Starting Work
+1. Open browser if not already open
+2. Navigate to the target site
+3. Wait for page to load completely
+4. **Take a screenshot after navigation**
+
+### Finding Elements
+- Use natural descriptions first
+- Fall back to text matching if needed
+- Never expose CSS selectors to users
+- **Take a screenshot when you find important elements**
+
+### Form Filling
+1. Find all form fields first
+2. **Take a screenshot of the empty form**
+3. Generate appropriate values using user context
+4. Fill fields in logical order
+5. **Take a screenshot after filling**
+6. Validate before submission
+7. **Take a screenshot after submission**
+
+### Completing Tasks
+- **Take screenshots at each major step**
+- Screenshots are saved automatically in the screenshots folder
+- Always close browser when done
+- Return clear summaries of what was accomplished
+
+## Common Workflows
+
+### Login Flow
+When you encounter a login page or need authentication:
+
+**If you have credentials from user:**
+1. Navigate to site
+2. Find and click login/sign in
+3. Fill credentials
+4. Submit and verify success
+
+**If you DON'T have credentials (most cases):**
+1. Navigate to the login page
+2. **Use `wait_for_manual_login("Site Name")` to pause**
+3. User will login manually in the browser
+4. User types 'yes' when done
+5. Continue with the task
+
+**Profile Persistence:**
+- Your browser profile saves cookies/sessions automatically
+- After first manual login, future runs will stay logged in
+- No need to login again until cookies expire
+
+### Form Submission
+1. Identify all required fields
+2. Generate appropriate values
+3. Fill and validate
+4. Submit and confirm
+
+### Information Extraction
+1. Navigate to target page
+2. Wait for content to load
+3. Extract relevant data
+4. Format and return results
+
+## Response Format
+
+Keep responses concise and informative:
+
+✅ **Good**: "Clicked the login button and filled in your email."
+
+❌ **Bad**: "I executed a click action on the element with selector #login-btn at coordinates (234, 456) and then performed a fill operation on the input element..."
+
+## Important Behaviors
+
+### Always
+- Report actions as you take them
+- Use natural language descriptions
+- Handle common scenarios automatically
+- Close resources when finished
+
+### Never
+- Ask for CSS selectors
+- Expose technical details unnecessarily
+- Leave browser open after task completion
+- Give up without trying alternatives
+
+## How Element Finding Works
+
+When you use `click("the login button")` or `type_text("the email field", "user@example.com")`:
+
+1. **System extracts all interactive elements** with their positions and text
+2. **You SELECT from indexed list** (by index), never generate CSS
+3. **Pre-built locators are used** - guaranteed to work
+
+### Examples
+
+**Clicking by text:**
+```
+User: "Click on Ryan Tan KK"
+System shows: [0] a "Home" [1] a "Priyanshu Mishra" [2] a "Ryan Tan KK"
+You select: index=2 (exact text match)
+```
+
+**Clicking by purpose:**
+```
+User: "Click the login button"
+System shows: [0] a "Home" [1] button "Sign In" [2] input placeholder="Email"
+You select: index=1 (Sign In = login button semantically)
+```
+
+**Clicking by position:**
+```
+User: "Click the first conversation"
+System shows: [0] input "Search" [1] a "John Doe" pos=(100,150) [2] a "Jane Smith" pos=(100,230)
+You select: index=1 (first conversation by vertical position)
+```
+
+The key insight: **You match descriptions to indexed elements, never generate CSS selectors.**
+
+## Error Handling
+
+When encountering errors:
+1. Try alternative approaches
+2. Explain the issue simply
+3. Suggest next steps
+4. Ask for clarification only when necessary
 
-You are a browser automation assistant that understands natural language requests for browser automation including navigation, interaction, screenshots, and debugging.
+## Task Completion
 
-## Your Available Functions
-
-### Navigation & State
-- `navigate_to(url)` - Navigate to any website
-- `get_current_url()` - Get the current page URL
-- `get_current_page_html()` - Get HTML content of current page
-- `wait(seconds)` - Wait for specified seconds (useful after navigation or clicks)
-
-### Viewport & Display
-- `set_viewport(width, height)` - Set custom viewport dimensions
-- `screenshot_with_iphone_viewport(url, path)` - Take screenshot with iPhone size (390x844)
-- `screenshot_with_ipad_viewport(url, path)` - Take screenshot with iPad size (768x1024)
-- `screenshot_with_desktop_viewport(url, path)` - Take screenshot with desktop size (1920x1080)
-- `take_screenshot(url, path, width, height, full_page)` - Take screenshot with all options
-
-### Interaction
-- `click_element_by_description(description)` - Click elements using natural language (e.g., "the login button", "menu icon")
-
-### Debugging
-- `get_debug_trace()` - Get execution trace when debugging issues
-
-## Understanding Requests
-
-Parse natural language flexibly. Use sensible defaults when details aren't specified:
-- If no path is given, use the default (screenshots are automatically saved to a temporary folder)
-- Only ask for clarification if truly necessary
-
-Users might say:
-- "screenshot localhost:3000"
-- "take a screenshot of example.com"
-- "capture google.com and save it to /tmp/test.png"
-- "screenshot the homepage with iPhone size"
-- "grab a pic of localhost:3000/api"
-
-## Choosing the Right Tool
-
-Based on viewport requirements:
-- If user mentions "iPhone" or "mobile" → use `screenshot_with_iphone_viewport`
-- If user mentions "iPad" or "tablet" → use `screenshot_with_ipad_viewport`
-- If user mentions "desktop" or "full" → use `screenshot_with_desktop_viewport`
-- For custom sizes or default → use `take_screenshot` with appropriate width/height
-
-## Response and Error Handling
-
-Be concise and direct:
-- On success: Use ✅ and report the result
-- On error: Use ❌ and provide helpful context
-- When actions fail: Call `get_debug_trace()` to understand what went wrong
-- Be natural and helpful without over-explaining
-
-### Success Examples:
-- "✅ Navigated to example.com"
-- "✅ Clicked the login button" 
-- "✅ Screenshot saved: .tmp/screenshot_20240101_120000.png"
-- "✅ Viewport set to 768x1024"
-
-### Error Handling:
-When an action fails (timeout, element not found, etc.):
-1. Report the error clearly
-2. Use `get_debug_trace()` if the issue is unclear
-3. Suggest alternatives or next steps
-
-Example error responses:
-- "❌ Could not find 'submit button'. The element may not be visible or loaded yet."
-- "❌ Navigation timeout. The page took too long to load."
-- "❌ Click failed. Let me check the debug trace... [calls get_debug_trace()]"
-
-When inputs are ambiguous or missing, ask one targeted question at a time, such as:
-- "Which URL should I open?"
-- "Do you want full-page or just the current viewport?"
-- "What viewport size should I use (iPhone, iPad, desktop, or custom width x height)?"
-
-## Examples
-
-### Basic Navigation
-User: "go to example.com and get the HTML"
-→ navigate_to("example.com"), then get_current_page_html()
-
-User: "navigate to localhost:3000 and click the login button"
-→ navigate_to("localhost:3000"), then click_element_by_description("login button")
-
-### Screenshots
-User: "screenshot localhost:3000"
-→ take_screenshot(url="localhost:3000") # Path is optional
-
-User: "screenshot mobile localhost:3000"
-→ screenshot_with_iphone_viewport(url="localhost:3000")
-
-User: "set viewport to tablet size and take a screenshot"
-→ set_viewport(768, 1024), then take_screenshot(current_url)
-
-### Complex Workflows
-User: "go to example.com, click more info link, check if URL changed"
-→ navigate_to("example.com"), get_current_url(), click_element_by_description("more info link"), wait(2), get_current_url()
-
-User: "navigate to localhost:3000, click menu button, wait for sidebar, then screenshot"
-→ navigate_to("localhost:3000"), click_element_by_description("menu button"), wait(1), take_screenshot(current_url)
-
-### Debugging
-User: "why did the click fail?"
-→ get_debug_trace() # Shows execution history
-
-Remember: Chain functions logically, use wait() after navigation/clicks when needed, and call get_debug_trace() when debugging issues.
+A task is complete when:
+- The requested action has been performed
+- Results have been extracted/saved
+- Browser has been closed (unless ongoing session)
+- User has been informed of the outcome
+
+Remember: You make web automation feel natural and effortless. Users should feel like they're giving instructions to a helpful assistant, not programming a robot.

connectonion/cli/browser_agent/prompts/element_matcher.md

@@ -0,0 +1,59 @@
+# Element Matcher
+
+You are an element matcher. Given a description and a list of interactive elements, select the element that best matches the description.
+
+## Examples
+
+### Example 1: Semantic matching
+DESCRIPTION: "the login button"
+ELEMENTS:
+[0] a "Home" pos=(50,20)
+[1] button "Sign In" pos=(900,20)
+[2] input placeholder="Email" pos=(400,300)
+
+Answer: index=1, reasoning="Sign In is the login button"
+
+### Example 2: Exact text match
+DESCRIPTION: "Ryan Tan KK"
+ELEMENTS:
+[0] div "Messages" pos=(0,100)
+[1] a "Priyanshu Mishra" pos=(100,200)
+[2] a "Ryan Tan KK" pos=(100,280)
+[3] a "Sijin Wang" pos=(100,360)
+
+Answer: index=2, reasoning="Exact text match for Ryan Tan KK"
+
+### Example 3: Position-based matching
+DESCRIPTION: "the first conversation"
+ELEMENTS:
+[0] input placeholder="Search" pos=(100,50)
+[1] a "John Doe Last message preview..." pos=(100,150)
+[2] a "Jane Smith Another message..." pos=(100,230)
+
+Answer: index=1, reasoning="First conversation in the list by position"
+
+### Example 4: Type + attribute matching
+DESCRIPTION: "email field"
+ELEMENTS:
+[0] button "Submit" pos=(400,500)
+[1] input placeholder="Enter your email" pos=(400,300) type=email
+[2] input placeholder="Password" pos=(400,380) type=password
+
+Answer: index=1, reasoning="Input with email type and email-related placeholder"
+
+## Your Task
+
+DESCRIPTION: "{description}"
+
+INTERACTIVE ELEMENTS:
+{element_list}
+
+Select the element index that best matches the description.
+
+Consider:
+- Text content matches (exact or partial)
+- Element type (button, link, input, etc.)
+- Position on page (first, second, top, bottom)
+- Semantic meaning (login=Sign In, search=magnifying glass)
+
+Return the index of the best matching element.

connectonion/cli/browser_agent/prompts/form_filler.md

@@ -0,0 +1,19 @@
+# Form Filler
+
+Generate appropriate form values based on the user information provided.
+
+## User Info
+{user_info}
+
+## Form Fields
+{field_descriptions}
+
+## Instructions
+
+For each form field, generate an appropriate value based on:
+- The field type (text, email, password, etc.)
+- The field label and name
+- Whether it's required
+- The user info provided
+
+Return a dictionary with field names as keys and appropriate values.

connectonion/cli/browser_agent/prompts/scroll_strategy.md

@@ -0,0 +1,36 @@
+# Scroll Strategy
+
+Analyze this webpage and determine the BEST way to scroll "{description}".
+
+## Scrollable Elements Found
+{scrollable_elements}
+
+## Simplified HTML (first 5000 chars)
+{simplified_html}
+
+## Instructions
+
+Return:
+
+1. **method**: "window" | "element" | "container"
+
+2. **selector**: CSS selector (empty if method is "window")
+
+3. **javascript**: Complete IIFE that scrolls ONE iteration:
+```javascript
+(() => {{
+  const el = document.querySelector('.selector');
+  if (el) el.scrollTop += 1000;
+  return {{success: true}};
+}})()
+```
+
+4. **explanation**: Brief reason
+
+## Common Patterns
+
+- Gmail/email lists: Scroll the container with overflow:auto, NOT window
+- Social feeds (Twitter, LinkedIn): Often scroll the main feed container
+- Regular pages: Use window.scrollBy(0, 1000)
+
+User wants to scroll: "{description}"

connectonion/cli/browser_agent/scripts/extract_elements.js

@@ -0,0 +1,126 @@
+/**
+ * Extract interactive elements from the page with injected IDs.
+ *
+ * Inspired by browser-use (https://github.com/browser-use/browser-use).
+ *
+ * This script:
+ * 1. Finds all interactive elements (buttons, links, inputs, etc.)
+ * 2. Injects a unique `data-browser-agent-id` attribute into each
+ * 3. Returns element data with bounding boxes for LLM matching
+ */
+(() => {
+    const results = [];
+    let index = 0;
+
+    // Interactive element types
+    const INTERACTIVE_TAGS = new Set([
+        'a', 'button', 'input', 'select', 'textarea', 'label',
+        'details', 'summary', 'dialog'
+    ]);
+
+    const INTERACTIVE_ROLES = new Set([
+        'button', 'link', 'menuitem', 'menuitemcheckbox', 'menuitemradio',
+        'option', 'radio', 'switch', 'tab', 'checkbox', 'textbox',
+        'searchbox', 'combobox', 'listbox', 'slider', 'spinbutton'
+    ]);
+
+    // Check if element is visible
+    function isVisible(el) {
+        const style = window.getComputedStyle(el);
+        if (style.display === 'none') return false;
+        if (style.visibility === 'hidden') return false;
+        if (parseFloat(style.opacity) === 0) return false;
+
+        // Skip visually-hidden accessibility elements
+        const className = el.className || '';
+        if (typeof className === 'string' &&
+            (className.includes('visually-hidden') ||
+             className.includes('sr-only') ||
+             className.includes('screen-reader'))) {
+            return false;
+        }
+
+        const rect = el.getBoundingClientRect();
+        if (rect.width === 0 || rect.height === 0) return false;
+
+        // Skip elements that are clipped/hidden with CSS tricks
+        if (rect.width < 2 || rect.height < 2) return false;
+
+        // Check if in viewport (with some margin)
+        const margin = 100;
+        if (rect.bottom < -margin) return false;
+        if (rect.top > window.innerHeight + margin) return false;
+        if (rect.right < -margin) return false;
+        if (rect.left > window.innerWidth + margin) return false;
+
+        return true;
+    }
+
+    // Get clean text content
+    function getText(el) {
+        // For inputs, get value or placeholder
+        if (el.tagName === 'INPUT' || el.tagName === 'TEXTAREA') {
+            return el.value || el.placeholder || '';
+        }
+        // For other elements, get inner text
+        const text = el.innerText || el.textContent || '';
+        return text.trim().replace(/\s+/g, ' ').substring(0, 80);
+    }
+
+    // Process all elements
+    document.querySelectorAll('*').forEach(el => {
+        const tag = el.tagName.toLowerCase();
+        const role = el.getAttribute('role');
+
+        // Check if interactive
+        const isInteractiveTag = INTERACTIVE_TAGS.has(tag);
+        const isInteractiveRole = role && INTERACTIVE_ROLES.has(role);
+        const isClickable = window.getComputedStyle(el).cursor === 'pointer';
+        const hasTabIndex = el.hasAttribute('tabindex') && el.tabIndex >= 0;
+        const hasClickHandler = el.onclick !== null || el.hasAttribute('onclick');
+
+        if (!isInteractiveTag && !isInteractiveRole && !isClickable &&
+            !hasTabIndex && !hasClickHandler) {
+            return;
+        }
+
+        // Skip hidden inputs
+        if (tag === 'input' && el.type === 'hidden') return;
+
+        // Skip empty elements with no text or useful attributes
+        const text = getText(el);
+        const ariaLabel = el.getAttribute('aria-label');
+        const placeholder = el.placeholder;
+        if (!text && !ariaLabel && !placeholder && tag !== 'input') return;
+
+        // Skip very small elements (likely icons)
+        const rect = el.getBoundingClientRect();
+        if (rect.width < 20 && rect.height < 20 && !text) return;
+
+        // Check visibility
+        if (!isVisible(el)) return;
+
+        // INJECT a unique ID attribute for reliable location
+        const highlightId = String(index);
+        el.setAttribute('data-browser-agent-id', highlightId);
+
+        results.push({
+            index: index++,
+            tag: tag,
+            text: text,
+            role: role,
+            aria_label: el.getAttribute('aria-label'),
+            placeholder: el.placeholder || null,
+            input_type: el.type || null,
+            href: (tag === 'a' && el.href) ? el.href.substring(0, 100) : null,
+            x: Math.round(rect.x),
+            y: Math.round(rect.y),
+            width: Math.round(rect.width),
+            height: Math.round(rect.height),
+            // Use injected attribute as locator - guaranteed to work!
+            locator: `[data-browser-agent-id="${highlightId}"]`
+        });
+    });
+
+    return results;
+})()

connectonion/cli/browser_agent/scroll.py

@@ -0,0 +1,137 @@
+"""
+Unified scroll module - AI-powered with fallback strategies.
+
+Usage:
+    from scroll import scroll
+    result = scroll(page, take_screenshot, times=5, description="the email list")
+"""
+from pathlib import Path
+from pydantic import BaseModel
+from connectonion import llm_do
+import time
+
+_PROMPT = (Path(__file__).parent / "prompts" / "scroll_strategy.md").read_text()
+
+
+class ScrollStrategy(BaseModel):
+    method: str  # "window", "element", "container"
+    selector: str
+    javascript: str
+    explanation: str
+
+
+def scroll(page, take_screenshot, times: int = 5, description: str = "the main content area") -> str:
+    """Universal scroll with AI strategy and fallback.
+
+    Tries: AI-generated → Element scroll → Page scroll
+    Verifies success with screenshot comparison.
+    """
+    if not page:
+        return "Browser not open"
+
+    timestamp = int(time.time())
+    before = f"scroll_before_{timestamp}.png"
+    take_screenshot(path=before)
+
+    strategies = [
+        ("AI strategy", lambda: _ai_scroll(page, times, description)),
+        ("Element scroll", lambda: _element_scroll(page, times)),
+        ("Page scroll", lambda: _page_scroll(page, times)),
+    ]
+
+    for name, execute in strategies:
+        print(f"  Trying: {name}...")
+        try:
+            execute()
+            time.sleep(0.5)
+            after = f"scroll_after_{timestamp}.png"
+            take_screenshot(path=after)
+
+            if _screenshots_different(before, after):
+                print(f"  ✅ {name} worked")
+                return f"Scrolled using {name}"
+            print(f"  ⚠️ {name} didn't change content")
+            before = after
+        except Exception as e:
+            print(f"  ❌ {name} failed: {e}")
+
+    return "All scroll strategies failed"
+
+
+def _ai_scroll(page, times: int, description: str):
+    """AI-generated scroll strategy."""
+    scrollable = page.evaluate("""
+        (() => {
+            return Array.from(document.querySelectorAll('*'))
+                .filter(el => {
+                    const s = window.getComputedStyle(el);
+                    return (s.overflow === 'auto' || s.overflowY === 'scroll') &&
+                           el.scrollHeight > el.clientHeight;
+                })
+                .slice(0, 3)
+                .map(el => ({tag: el.tagName, classes: el.className, id: el.id}));
+        })()
+    """)
+
+    html = page.evaluate("""
+        (() => {
+            const c = document.body.cloneNode(true);
+            c.querySelectorAll('script,style,img,svg').forEach(e => e.remove());
+            return c.innerHTML.substring(0, 5000);
+        })()
+    """)
+
+    strategy = llm_do(
+        _PROMPT.format(description=description, scrollable_elements=scrollable, simplified_html=html),
+        output=ScrollStrategy,
+        model="co/gemini-2.5-flash",
+        temperature=0.1
+    )
+    print(f"    AI: {strategy.explanation}")
+
+    for _ in range(times):
+        page.evaluate(strategy.javascript)
+        time.sleep(1)
+
+
+def _element_scroll(page, times: int):
+    """Scroll first scrollable element found."""
+    for _ in range(times):
+        page.evaluate("""
+            (() => {
+                const el = Array.from(document.querySelectorAll('*')).find(e => {
+                    const s = window.getComputedStyle(e);
+                    return (s.overflow === 'auto' || s.overflowY === 'scroll') &&
+                           e.scrollHeight > e.clientHeight;
+                });
+                if (el) el.scrollTop += 1000;
+            })()
+        """)
+        time.sleep(0.8)
+
+
+def _page_scroll(page, times: int):
+    """Scroll window."""
+    for _ in range(times):
+        page.evaluate("window.scrollBy(0, 1000)")
+        time.sleep(0.8)
+
+
+def _screenshots_different(file1: str, file2: str) -> bool:
+    """Compare screenshots using PIL pixel difference."""
+    try:
+        from PIL import Image
+        import os
+
+        img1 = Image.open(os.path.join("screenshots", file1)).convert('RGB')
+        img2 = Image.open(os.path.join("screenshots", file2)).convert('RGB')
+
+        diff = sum(
+            abs(a - b)
+            for p1, p2 in zip(img1.getdata(), img2.getdata())
+            for a, b in zip(p1, p2)
+        )
+        threshold = img1.size[0] * img1.size[1] * 3 * 0.01  # 1%
+        return diff > threshold
+    except Exception:
+        return True  # Assume different if comparison fails

{connectonion-0.6.1.dist-info → connectonion-0.6.2.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: connectonion
-Version: 0.6.1
+Version: 0.6.2
 Summary: A simple Python framework for creating AI agents with behavior tracking
 Project-URL: Homepage, https://github.com/openonion/connectonion
 Project-URL: Documentation, https://docs.connectonion.com