v16.ai 0.2.15 → 0.3.0

package/BROWSER_AUTOMATION.md

@@ -0,0 +1,513 @@
+# Browser Automation & Workflow System
+
+The V16 local agent now includes powerful browser automation capabilities with secure credential management, workflow automation, web search, and email checking.
+
+## Features
+
+### 1. **Secure Credential Vault**
+- AES-256-GCM encryption for passwords and API keys
+- Stores credentials locally on your machine
+- Master key protection
+- Domain-based credential lookup
+
+### 2. **Workflow Automation**
+- Step-by-step browser automation
+- Credential injection into forms
+- Screenshot capture after each step
+- Error handling and recovery
+
+### 3. **Web Search**
+- Google and DuckDuckGo integration
+- Result extraction and parsing
+- Information extraction from web pages
+
+### 4. **Email Checking**
+- Gmail and Outlook support
+- Check inbox, search emails, find unread
+- Read specific email content
+
+### 5. **Session Persistence**
+- Save browser sessions (cookies, localStorage)
+- Restore sessions to avoid repeated logins
+- Domain-based session management
+
+## API Endpoints
+
+### Credential Management
+
+```bash
+# Store a credential
+POST /api/local-agent/credentials
+{
+  "name": "My Gmail Password",
+  "type": "password",
+  "domain": "gmail.com",
+  "username": "user@gmail.com",
+  "value": "secret123"
+}
+
+# List all credentials
+GET /api/local-agent/credentials
+
+# Get specific credential
+GET /api/local-agent/credentials/:id
+
+# Delete credential
+DELETE /api/local-agent/credentials/:id
+
+# Find credentials by domain
+GET /api/local-agent/credentials/domain/gmail.com
+```
+
+### Workflow Management
+
+```bash
+# Create workflow
+POST /api/local-agent/workflows
+{
+  "name": "Gmail Login",
+  "description": "Automated Gmail login",
+  "steps": [
+    {
+      "action": "navigate",
+      "value": "https://mail.google.com"
+    },
+    {
+      "action": "type",
+      "selector": "input[type='email']",
+      "credentialId": "credential-id-here"
+    },
+    {
+      "action": "click",
+      "selector": "button[type='submit']"
+    }
+  ],
+  "credentialIds": ["credential-id-here"]
+}
+
+# List workflows
+GET /api/local-agent/workflows
+
+# Execute workflow
+POST /api/local-agent/workflows/:id/execute
+```
+
+### Web Search
+
+```bash
+# Search the web
+POST /api/local-agent/search
+{
+  "query": "latest news on AI",
+  "maxResults": 10,
+  "engine": "google" // or "duckduckgo" or "auto"
+}
+
+# Extract information
+POST /api/local-agent/search/extract
+{
+  "query": "weather in New York",
+  "task": "Get current temperature"
+}
+```
+
+### Email Checking
+
+```bash
+# Check emails
+POST /api/local-agent/email/check
+{
+  "provider": "gmail", // or "outlook"
+  "maxEmails": 20
+}
+
+# Search emails
+POST /api/local-agent/email/search
+{
+  "provider": "gmail",
+  "query": "from:someone@example.com"
+}
+
+# Get unread emails
+POST /api/local-agent/email/unread
+{
+  "provider": "gmail"
+}
+
+# Read specific email
+POST /api/local-agent/email/read
+{
+  "provider": "gmail",
+  "emailIdentifier": "subject query or from:email"
+}
+```
+
+### Session Management
+
+```bash
+# Save current browser session
+POST /api/local-agent/sessions/save
+{
+  "domain": "gmail.com",
+  "name": "Gmail Session"
+}
+
+# Restore session
+POST /api/local-agent/sessions/restore
+{
+  "sessionId": "session-credential-id"
+}
+
+# List sessions
+GET /api/local-agent/sessions
+
+# Delete session
+DELETE /api/local-agent/sessions/:id
+```
+
+## Example Workflows
+
+### Example 1: Gmail Login and Check Unread
+
+```json
+{
+  "name": "Gmail Morning Check",
+  "description": "Login to Gmail and check unread emails",
+  "steps": [
+    {
+      "stepId": "step-1",
+      "action": "navigate",
+      "value": "https://mail.google.com"
+    },
+    {
+      "stepId": "step-2",
+      "action": "wait",
+      "waitMs": 2000
+    },
+    {
+      "stepId": "step-3",
+      "action": "screenshot"
+    }
+  ],
+  "credentialIds": []
+}
+```
+
+### Example 2: LinkedIn Profile Update
+
+```json
+{
+  "name": "Update LinkedIn Profile",
+  "description": "Navigate to LinkedIn and update profile headline",
+  "steps": [
+    {
+      "stepId": "step-1",
+      "action": "navigate",
+      "value": "https://www.linkedin.com/in/me/"
+    },
+    {
+      "stepId": "step-2",
+      "action": "wait",
+      "waitMs": 2000
+    },
+    {
+      "stepId": "step-3",
+      "action": "click",
+      "selector": "button[aria-label='Edit intro']"
+    },
+    {
+      "stepId": "step-4",
+      "action": "wait",
+      "waitMs": 1000
+    },
+    {
+      "stepId": "step-5",
+      "action": "type",
+      "selector": "input[name='headline']",
+      "value": "AI Engineer | Building the Future"
+    },
+    {
+      "stepId": "step-6",
+      "action": "click",
+      "selector": "button[type='submit']"
+    },
+    {
+      "stepId": "step-7",
+      "action": "screenshot"
+    }
+  ],
+  "credentialIds": []
+}
+```
+
+### Example 3: Research Automation
+
+```json
+{
+  "name": "Market Research",
+  "description": "Search for competitor information and extract data",
+  "steps": [
+    {
+      "stepId": "step-1",
+      "action": "navigate",
+      "value": "https://www.google.com"
+    },
+    {
+      "stepId": "step-2",
+      "action": "type",
+      "selector": "textarea[name='q']",
+      "value": "best AI coding assistants 2026"
+    },
+    {
+      "stepId": "step-3",
+      "action": "click",
+      "selector": "input[type='submit']"
+    },
+    {
+      "stepId": "step-4",
+      "action": "wait",
+      "waitMs": 3000
+    },
+    {
+      "stepId": "step-5",
+      "action": "screenshot"
+    },
+    {
+      "stepId": "step-6",
+      "action": "extract",
+      "selector": "#search"
+    }
+  ],
+  "credentialIds": []
+}
+```
+
+### Example 4: Form Filling with Credentials
+
+```json
+{
+  "name": "Automated Form Submission",
+  "description": "Fill out a contact form with saved credentials",
+  "steps": [
+    {
+      "stepId": "step-1",
+      "action": "navigate",
+      "value": "https://example.com/contact"
+    },
+    {
+      "stepId": "step-2",
+      "action": "type",
+      "selector": "input[name='name']",
+      "credentialId": "name-credential-id"
+    },
+    {
+      "stepId": "step-3",
+      "action": "type",
+      "selector": "input[name='email']",
+      "credentialId": "email-credential-id"
+    },
+    {
+      "stepId": "step-4",
+      "action": "type",
+      "selector": "textarea[name='message']",
+      "value": "Hello, I'm interested in your product."
+    },
+    {
+      "stepId": "step-5",
+      "action": "click",
+      "selector": "button[type='submit']"
+    },
+    {
+      "stepId": "step-6",
+      "action": "wait",
+      "waitMs": 2000
+    },
+    {
+      "stepId": "step-7",
+      "action": "screenshot"
+    }
+  ],
+  "credentialIds": ["name-credential-id", "email-credential-id"]
+}
+```
+
+## Session Persistence Best Practices
+
+### Saving a Session
+
+After logging in to a website manually (e.g., Gmail), save the session:
+
+```bash
+curl -X POST http://localhost:3000/api/local-agent/sessions/save \
+  -H "Content-Type: application/json" \
+  -H "x-user-id: your-user-id" \
+  -d '{
+    "domain": "gmail.com",
+    "name": "My Gmail Session"
+  }'
+```
+
+### Restoring a Session
+
+Before running a workflow, restore the saved session:
+
+```bash
+curl -X POST http://localhost:3000/api/local-agent/sessions/restore \
+  -H "Content-Type: application/json" \
+  -H "x-user-id: your-user-id" \
+  -d '{
+    "sessionId": "session-credential-id"
+  }'
+```
+
+## Security Considerations
+
+1. **Credentials are stored locally** on your machine in `~/.v16/vault/`
+2. **AES-256-GCM encryption** protects all sensitive data
+3. **Master key** is stored with restricted permissions (0600)
+4. **Never share** your vault files or master key
+5. **Sessions** include cookies and storage - treat them as sensitive
+
+## Workflow Actions
+
+Available actions in workflows:
+
+- `navigate`: Navigate to a URL
+  - `value`: URL to navigate to
+
+- `click`: Click an element
+  - `selector`: CSS selector for element
+
+- `type`: Type text into an input
+  - `selector`: CSS selector for input
+  - `value`: Text to type (or use `credentialId` to inject from vault)
+  - `credentialId`: (optional) ID of credential to use as value
+
+- `wait`: Wait for a duration
+  - `waitMs`: Milliseconds to wait
+
+- `screenshot`: Take a screenshot
+  - No parameters needed
+
+- `extract`: Extract text from an element
+  - `selector`: CSS selector for element
+
+## Error Handling
+
+Workflows stop on the first error. Each step returns:
+
+```json
+{
+  "stepId": "step-1",
+  "success": true,
+  "action": "navigate",
+  "result": {
+    "url": "https://example.com",
+    "title": "Example Domain"
+  },
+  "screenshot": "data:image/png;base64,...",
+  "duration": 1234
+}
+```
+
+## Tips for Reliable Automation
+
+1. **Use persistent browser mode** - sessions survive restarts
+2. **Add wait steps** after navigation and clicks
+3. **Save sessions** after manual logins
+4. **Use domain-specific credentials** for easy lookup
+5. **Test workflows** with screenshots after critical steps
+6. **Handle errors** by checking step results
+7. **Update selectors** if websites change
+
+## Development Mode
+
+For local testing without authentication:
+
+```bash
+# Start local agent
+v16 connect --dev
+
+# Make requests with x-user-id header
+curl -X POST http://localhost:3000/api/local-agent/credentials \
+  -H "Content-Type: application/json" \
+  -H "x-user-id: test-user" \
+  -d '{"name": "Test", "type": "password", "value": "secret"}'
+```
+
+## Troubleshooting
+
+### Browser not initializing
+- Check if Chrome is installed
+- Try `browser-init` command with different modes: `existing`, `persistent`, or `fresh`
+
+### Credentials not found
+- Ensure credential vault is initialized
+- Check credential ID is correct
+- Verify domain matches
+
+### Workflow execution fails
+- Check each step result for errors
+- Verify selectors are correct (websites change!)
+- Add wait steps between actions
+- Use screenshots to debug
+
+### Session restore not working
+- Ensure you navigated to the same domain
+- Check if cookies are still valid
+- Some sites may require re-login after time
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     Backend API                              │
+│  /api/local-agent/* endpoints                                │
+└───────────────────────┬─────────────────────────────────────┘
+                        │ Socket.io
+                        │
+┌───────────────────────▼─────────────────────────────────────┐
+│                  Local Agent (CLI)                           │
+│                                                               │
+│  ┌─────────────────┐  ┌──────────────────┐                  │
+│  │ Credential Vault│  │ Workflow Engine  │                  │
+│  │ (AES-256-GCM)  │  │ (Step Executor)  │                  │
+│  └─────────────────┘  └──────────────────┘                  │
+│                                                               │
+│  ┌─────────────────┐  ┌──────────────────┐                  │
+│  │   Web Search    │  │  Email Checker   │                  │
+│  │  (Google/DDG)   │  │ (Gmail/Outlook)  │                  │
+│  └─────────────────┘  └──────────────────┘                  │
+│                                                               │
+│  ┌──────────────────────────────────────┐                   │
+│  │     Puppeteer Browser Automation     │                   │
+│  │  (Existing/Persistent/Fresh modes)   │                   │
+│  └──────────────────────────────────────┘                   │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Future Enhancements
+
+- [ ] Multi-step transaction rollback
+- [ ] Workflow scheduling and cron
+- [ ] Visual workflow builder
+- [ ] Playwright support (recommended by 2026 research)
+- [ ] Cloud session storage
+- [ ] Workflow templates marketplace
+- [ ] Integration with password managers
+- [ ] 2FA automation support
+- [ ] Headless browser optimization
+- [ ] Session validation before workflow execution
+
+## Contributing
+
+When adding new workflow actions:
+
+1. Add the action type to `WorkflowStep` interface
+2. Implement the action in `WorkflowAutomation.executeStep()`
+3. Update this documentation
+4. Add example workflows
+
+## License
+
+Part of V16 General AI Agent platform.

package/CHANGELOG.md

@@ -0,0 +1,164 @@
+# Changelog
+
+All notable changes to the V16 Local Agent will be documented in this file.
+
+## [0.3.0] - 2026-01-30
+
+### Added - Major Browser Automation Update 🎉
+
+#### Secure Credential Vault
+- **AES-256-GCM encryption** for passwords, API keys, and sensitive data
+- Local-only storage in `~/.v16/vault/`
+- Domain-based credential lookup
+- Master key protection with file permissions (0600)
+- CRUD API endpoints for credential management
+
+#### Workflow Automation System
+- Multi-step browser workflow execution
+- Support for 6 workflow actions: navigate, click, type, wait, screenshot, extract
+- Credential injection into form fields
+- Step-by-step execution with error handling
+- Screenshot capture after each action
+- Workflow storage and retrieval from encrypted vault
+
+#### Web Search Integration
+- **Google Search** - Programmatic search with result extraction
+- **DuckDuckGo Search** - Privacy-focused alternative
+- Automatic engine fallback (Google → DuckDuckGo)
+- Information extraction from top search results
+- Screenshot capture of search results
+
+#### Email Checking
+- **Gmail support** - Check inbox, search, find unread
+- **Outlook/Hotmail support** - Full inbox access
+- Email metadata extraction (sender, subject, snippet, date, unread status)
+- Email search functionality
+- Read specific email content
+
+#### Browser Session Persistence
+- Save browser sessions (cookies, localStorage, sessionStorage)
+- Restore sessions to avoid repeated logins
+- Domain-based session management
+- Session storage in encrypted credential vault
+
+#### API Endpoints
+New endpoints added to `/api/local-agent/`:
+- `POST /credentials` - Store encrypted credentials
+- `GET /credentials` - List all credentials
+- `GET /credentials/:id` - Get specific credential
+- `DELETE /credentials/:id` - Delete credential
+- `GET /credentials/domain/:domain` - Find credentials by domain
+- `POST /workflows` - Create workflow
+- `GET /workflows` - List workflows
+- `POST /workflows/:id/execute` - Execute workflow
+- `POST /search` - Web search
+- `POST /search/extract` - Extract information from search
+- `POST /email/check` - Check emails
+- `POST /email/unread` - Get unread emails
+- `POST /sessions/save` - Save browser session
+- `POST /sessions/restore` - Restore session
+
+#### Documentation
+- **BROWSER_AUTOMATION.md** - Complete documentation for all new features
+- **QUICKSTART.md** - 5-minute quick start guide with curl examples
+- **examples.ts** - 10+ ready-to-use workflow templates
+- Updated README.md with new capabilities
+
+### Changed
+- Updated package description to reflect new capabilities
+- Added new keywords: workflow, credential-vault, password-manager, email-checker, web-scraping, form-automation
+- Version bump from 0.2.15 to 0.3.0
+
+### Dependencies
+- Added `axios` ^1.13.4 for HTTP requests
+
+### Security
+- All credentials encrypted with AES-256-GCM
+- Master key stored with restricted permissions (0600)
+- Credentials never leave local machine
+- Session data encrypted before storage
+
+## [0.2.15] - Previous version
+
+### Features
+- Desktop control (mouse, keyboard, screenshots)
+- Browser automation with Puppeteer
+- Three-tier browser approach (Existing → Persistent → Fresh)
+- File system access
+- Terminal/PTY sessions
+- CLI command execution
+
+---
+
+## Upgrade Guide
+
+### From 0.2.x to 0.3.0
+
+No breaking changes. All existing functionality remains the same. New features are additive.
+
+To use the new features:
+
+1. **Update the package**:
+   ```bash
+   npm update -g v16.ai
+   ```
+
+2. **Credential vault is auto-initialized** when you connect:
+   ```bash
+   v16 connect --token YOUR_TOKEN
+   ```
+
+3. **Start using new endpoints** - see QUICKSTART.md for examples
+
+### Example Migration
+
+**Before (0.2.x)**: Manual browser automation
+```bash
+curl -X POST /api/local-agent/browser/navigate \
+  -d '{"url": "https://example.com"}'
+curl -X POST /api/local-agent/browser/type \
+  -d '{"selector": "input", "text": "password123"}'
+```
+
+**After (0.3.0)**: Automated workflow with credentials
+```bash
+# 1. Store credential once
+curl -X POST /api/local-agent/credentials \
+  -d '{"name": "My Password", "type": "password", "value": "password123"}'
+
+# 2. Create reusable workflow
+curl -X POST /api/local-agent/workflows \
+  -d '{
+    "name": "Login",
+    "steps": [
+      {"action": "navigate", "value": "https://example.com"},
+      {"action": "type", "selector": "input", "credentialId": "CRED_ID"}
+    ]
+  }'
+
+# 3. Execute workflow anytime
+curl -X POST /api/local-agent/workflows/WORKFLOW_ID/execute
+```
+
+---
+
+## Roadmap
+
+### Planned for 0.4.0
+- [ ] Playwright support (recommended by 2026 research)
+- [ ] Enhanced session validation
+- [ ] Workflow scheduling (cron)
+- [ ] 2FA automation support
+
+### Planned for 0.5.0
+- [ ] Visual workflow builder (web UI)
+- [ ] Workflow marketplace/templates
+- [ ] Cloud session sync (optional)
+- [ ] Password manager integrations
+
+---
+
+For more details, see:
+- [Quick Start Guide](./QUICKSTART.md)
+- [Browser Automation Documentation](./BROWSER_AUTOMATION.md)
+- [Example Workflows](./src/workflows/examples.ts)