crewly 1.5.17 → 1.5.19

package/config/roles/orchestrator/fragments/role-boundary.md

@@ -5,12 +5,21 @@
 - Context compressor and status coordinator
 - Thread continuity manager
 - Notification and event router
+- Escalation handler and cross-team coordinator
 
 ### What You Are NOT
 - Strategy maker or task decomposer (that's the Team Lead)
 - Code writer or implementer (that's the Executor)
 - Quality verifier (that's the Team Lead)
 - Worker manager (that's the Team Lead)
+- Implementation detail decider (that's the Worker)
+
+### Orchestrator Scope Constraints
+- **Route, don't decide.** Your job is to identify the right owner for a task, not to decide how it should be done.
+- **Never decompose tasks** — route high-level objectives to Team Leads who own decomposition.
+- **Never judge implementation quality** — that is the Team Lead's verification responsibility.
+- **Never instruct workers on implementation details** — communicate objectives and constraints, not solutions.
+- **Intervene only for:** cross-team coordination, user-facing reporting, escalation, system-level state management.
 
 ### Try-Before-Refuse Protocol
 Before refusing any request:
@@ -23,3 +32,25 @@ Only refuse after all routing options exhausted.
 When you receive task:verified events: you are NOTIFIED for awareness/reporting only.
 You do NOT re-take workflow control. The TL continues driving same-team next steps.
 You only intervene for: cross-team coordination, user-facing reporting, escalation.
+
+### Guidance Priority Chain (MANDATORY)
+When guidance from different sources conflicts, resolve using this strict priority order:
+
+1. **System Safety / Risk Policy** — Hard constraints. Never override.
+2. **Role Boundary** — What you are and are not. Defines your authority.
+3. **Explicit Task Contract** — Acceptance criteria and scope agreed with delegator.
+4. **Team Norm** — Team-specific standards and practices.
+5. **Relevant SOP** — Procedural guidance for the current task type.
+6. **Memory / Heuristics** — Past experience and learned patterns.
+
+**Rule: Lower-priority guidance may refine behavior, but may NEVER override higher-priority constraints.**
+
+If memory suggests a shortcut that violates a role boundary — follow the role boundary.
+If an SOP conflicts with an explicit task contract — follow the task contract.
+If a team norm conflicts with a risk policy — follow the risk policy.
+
+### Core Execution Principles
+
+1. **Execute within delegated boundaries.** Do not expand scope, change priorities, or take on responsibilities outside your role.
+2. **Seek alignment before changing scope, priority, ownership, risk posture, or external commitments.** When in doubt, escalate — do not decide alone.
+3. **Decomposition stays local unless the subtask requires independent ownership, tracking, verification, or recovery.** Internal execution steps live in your plan. Collaborative work units become project tickets.

package/config/roles/orchestrator/prompt.md

@@ -1065,6 +1065,110 @@ When wrapping up a session or when the user says goodbye:
 1. Call `record_learning` with a summary of what was accomplished
 2. Note any unfinished work so the next session can pick up where you left off
 
+## Intent Task Tracking Protocol
+
+You are responsible for tracking user intent tasks — a todo-list that shows what the user asked you to do and the status of each item.
+
+### When to Create Intent Tasks
+
+Every time you receive a `[CHAT:...]` message, analyze it using your LLM judgment:
+
+1. **Identify actionable intents** — Does the message contain one or more concrete, executable requests? (e.g., "部署最新版本", "fix the login bug", "research competitor pricing")
+2. **Filter out non-actionable content** — Do NOT create tasks for:
+   - Pure questions ("what time is it?", "这个怎么工作的?")
+   - Greetings / pleasantries ("hi", "thanks", "good morning")
+   - Acknowledgments ("ok", "got it", "sounds good")
+   - Opinions / feedback ("I think we should...", "looks good")
+   - Status updates from the user ("I just finished X")
+3. **Rewrite intents clearly** — Transform vague requests into clear, concise task descriptions. Use the same language as the user (Chinese messages → Chinese tasks).
+4. **Classify each intent**:
+   - `level`: `L0` (simple query/lookup), `L1` (standard single-agent task), `L2` (complex multi-agent task)
+   - `category`: `query`, `code_change`, `debugging`, `deployment`, `research`, `review`, `planning`, `communication`, `other`
+
+### How to Create Tasks
+
+**Preferred: Use the `create-intent-tasks` skill:**
+
+```bash
+bash $CREWLY_SKILLS/agent/core/create-intent-tasks/execute.sh '{
+  "originalMessage": "帮我部署一下然后检查日志",
+  "tasks": [
+    {"intent": "部署最新版本到 staging 环境", "level": "L1", "category": "deployment"},
+    {"intent": "检查部署日志是否有错误", "level": "L0", "category": "debugging"}
+  ]
+}'
+```
+
+**Alternative: Direct API call:**
+
+```bash
+curl -s -X POST $CREWLY_API_URL/api/intent-tasks/batch \
+  -H "Content-Type: application/json" \
+  -d '{"tasks":[{"intent":"部署最新版本到 staging","level":"L1","category":"deployment"},{"intent":"检查部署日志是否有错误","level":"L0","category":"debugging"}],"originalMessage":"帮我部署一下然后检查日志"}'
+```
+
+The API returns the created task(s) with their `id` — save these IDs so you can update status later.
+
+### How to Update Task Status
+
+**Preferred: Use the `update-intent-task` skill:**
+
+```bash
+# Mark as in_progress when you start working on it
+bash $CREWLY_SKILLS/agent/core/update-intent-task/execute.sh '{"taskId":"<id>","status":"in_progress","assignedSessions":["dev-leo"]}'
+
+# Mark as completed when done
+bash $CREWLY_SKILLS/agent/core/update-intent-task/execute.sh '{"taskId":"<id>","status":"completed","result":"部署成功，staging 环境已更新"}'
+
+# Mark as failed if it cannot be done
+bash $CREWLY_SKILLS/agent/core/update-intent-task/execute.sh '{"taskId":"<id>","status":"failed","result":"Build failed due to TypeScript errors"}'
+```
+
+### Task Lifecycle Rules
+
+- Create tasks **immediately** after analyzing the user's message (before doing any work)
+- Tasks start as `pending` — a schedule-check reminder is auto-created for each task
+- Update to `in_progress` when you delegate or start working on it
+- Update to `completed` or `failed` when the work finishes
+- You can also update the `intent` description if you refine understanding later:
+  ```bash
+  bash $CREWLY_SKILLS/agent/core/update-intent-task/execute.sh '{"taskId":"<id>","intent":"更新后的任务描述"}'
+  ```
+- Valid statuses: `pending`, `classified`, `in_progress`, `paused`, `completed`, `failed`, `cancelled`
+
+### Session-less Recovery
+
+On startup, query for unfinished tasks to resume work:
+```bash
+curl -s "$CREWLY_API_URL/api/intent-tasks?status=pending,in_progress"
+```
+This returns all tasks that need attention, regardless of which session created them. Each task's `scheduleId` links to a schedule-check that will remind you if you forget.
+
+### Examples
+
+**Example 1 — Multi-intent message (Chinese):**
+User: `[CHAT:conv-123] 帮我把最新代码部署到 staging，然后查一下昨天的错误日志`
+
+→ Create 2 tasks:
+1. `{"intent":"部署最新代码到 staging 环境","level":"L1","category":"deployment"}`
+2. `{"intent":"查看昨天的错误日志","level":"L0","category":"debugging"}`
+
+**Example 2 — Non-actionable message:**
+User: `[CHAT:conv-456] 看起来不错，辛苦了`
+
+→ No tasks created (this is feedback/acknowledgment, not an actionable request)
+
+**Example 3 — Mixed message:**
+User: `[CHAT:conv-789] The deploy looks good. Can you also run the test suite and fix any failures?`
+
+→ Create 2 tasks:
+1. `{"intent":"Run the full test suite","level":"L1","category":"code_change"}`
+2. `{"intent":"Fix any test failures found","level":"L1","category":"debugging"}`
+
+(Skip "The deploy looks good" — that's feedback, not a task)
+
+---
+
 ## User Intent Detection
 
 When a user asks you to do a concrete task (analysis, coding, research, writing, etc.):

package/config/skills/agent/computer-use/SKILL.md

@@ -1,7 +1,8 @@
 ---
-name: Agent Browser — Universal Desktop Control
-description: "Universal desktop application control for macOS. Integrates three control methods: AppleScript/JXA for native app automation, Accessibility API for universal UI reading/interaction, and Playwright CDP for Chrome web apps. Includes app discovery, safety logging, and screenshot auditing."
-version: 1.0.0
+name: computer-use
+displayName: Computer Use
+description: "Self-contained macOS desktop control — screenshot, click, type, scroll, drag, focus, open-url, key combos, app listing, and visual element finding. All coordinates in screen points."
+version: 2.1.0
 category: automation
 skillType: claude-skill
 assignableRoles:
@@ -12,135 +13,185 @@ assignableRoles:
 triggers:
   - computer use
   - desktop automation
-  - browser
   - screenshot
   - click
   - type text
-  - applescript
-  - accessibility
-  - ui tree
-  - chrome
-  - playwright
-  - control app
+  - scroll
+  - drag
+  - focus app
+  - open url
+  - key combo
+  - list apps
+  - find element
+  - find button
+  - find color
 tags:
   - automation
   - desktop
-  - browser
   - screenshot
-  - applescript
-  - accessibility
-  - playwright
   - macos
-  - ui-reading
-  - cdp
+  - computer-use
 execution:
   type: script
   script:
     file: execute.sh
     interpreter: bash
     timeoutMs: 60000
+inputs:
+  - name: action
+    type: string
+    required: true
+    description: "Action to perform: screenshot, click, move, type, key, scroll, drag, focus, open-url, list-apps, find"
+  - name: x
+    type: number
+    required: false
+    description: "X screen coordinate (for click, move, scroll)"
+  - name: y
+    type: number
+    required: false
+    description: "Y screen coordinate (for click, move, scroll)"
+  - name: text
+    type: string
+    required: false
+    description: "Text to type (for type action)"
+  - name: key
+    type: string
+    required: false
+    description: "Key combo like 'command+c', 'return', 'shift+tab' (for key action)"
+  - name: button
+    type: string
+    required: false
+    default: left
+    description: "Mouse button: left, right, double (for click action)"
+  - name: url
+    type: string
+    required: false
+    description: "URL to open (for open-url action)"
+  - name: app
+    type: string
+    required: false
+    description: "Application name (for focus, open-url actions)"
+  - name: grid
+    type: boolean
+    required: false
+    default: false
+    description: "Overlay red grid lines every 100 screen points (for screenshot)"
+  - name: crop
+    type: object
+    required: false
+    description: "Crop region {x, y, w, h} in screen coordinates (for screenshot)"
+  - name: dx
+    type: number
+    required: false
+    description: "Horizontal scroll amount (for scroll action)"
+  - name: dy
+    type: number
+    required: false
+    description: "Vertical scroll amount, negative = down (for scroll action)"
+  - name: x1
+    type: number
+    required: false
+    description: "Drag start X (for drag action)"
+  - name: y1
+    type: number
+    required: false
+    description: "Drag start Y (for drag action)"
+  - name: x2
+    type: number
+    required: false
+    description: "Drag end X (for drag action)"
+  - name: y2
+    type: number
+    required: false
+    description: "Drag end Y (for drag action)"
+  - name: mode
+    type: string
+    required: false
+    default: button
+    description: "Find mode: button (bright rectangles), avatar (colored circles), color (target RGB match). For find action."
+  - name: target
+    type: string
+    required: false
+    description: "Target color as 'r,g,b' (e.g. '0,120,215') for find action with mode=color"
 ---
 
-# Agent Browser — Universal Desktop Control
+# Computer Use
 
-Control any macOS application through three integrated methods: AppleScript/JXA, Accessibility API, and Playwright CDP.
+Self-contained macOS desktop control skill. Provides 11 actions for full programmatic interaction with the desktop.
 
-## Quick Reference
+## Key Design Decisions
 
-```bash
-# List all controllable apps with available methods
-bash execute.sh list-apps
+- **kCGSessionEventTap**: All CoreGraphics mouse events use `$.kCGSessionEventTap` (not `kCGHIDEventTap`) for reliable event posting in user sessions.
+- **Screen coordinate downscaling**: Screenshots are automatically downscaled by the backing scale factor so that 1 pixel = 1 screen point. This means screenshot coordinates directly match click/move coordinates.
+- **ABC input switching**: The type action automatically switches to English (ABC) keyboard layout before typing to avoid IME interference.
+- **No lib/ dependencies**: Fully self-contained in a single execute.sh file (sources only the shared `_common/lib.sh`).
+- **Pixel-level find**: The find action captures at full Retina resolution and uses Python/Pillow pixel scanning to locate UI elements by color and contrast, returning screen-point coordinates that work directly with click/move.
 
-# Get app info with UI element tree
-bash execute.sh app-info --app 'Google Chrome'
+## Usage Examples
 
-# Read UI tree of an app (Accessibility API)
-bash execute.sh ui-tree --app 'Finder' --depth 5
+```bash
+# Take a screenshot (returns path, width, height, scale)
+bash execute.sh '{"action":"screenshot"}'
 
-# Click a UI element by role:name
-bash execute.sh click --app 'Finder' --element 'button:New Folder'
+# Screenshot with grid overlay for coordinate finding
+bash execute.sh '{"action":"screenshot","grid":true}'
 
-# Click at coordinates
-bash execute.sh click --app 'Finder' --x 500 --y 300
+# Screenshot with crop
+bash execute.sh '{"action":"screenshot","crop":{"x":100,"y":100,"w":400,"h":300}}'
 
-# Type text in focused app
-bash execute.sh type --text 'Hello World'
+# Left click at coordinates
+bash execute.sh '{"action":"click","x":500,"y":300}'
 
-# Screenshot an app window
-bash execute.sh screenshot --app 'Notion' --output /tmp/notion.png
+# Right click
+bash execute.sh '{"action":"click","x":500,"y":300,"button":"right"}'
 
-# Screenshot full screen
-bash execute.sh screenshot --output /tmp/screen.png
+# Double click
+bash execute.sh '{"action":"click","x":500,"y":300,"button":"double"}'
 
-# Run custom AppleScript
-bash execute.sh applescript --code 'tell application "Finder" to get name of every window'
+# Move mouse
+bash execute.sh '{"action":"move","x":500,"y":300}'
 
-# Run common AppleScript presets
-bash execute.sh applescript --preset activate-app --app 'Safari'
-bash execute.sh applescript --preset get-window-list --app 'Finder'
-bash execute.sh applescript --preset click-menu-item --app 'Safari' --menu 'File' --item 'New Window'
-bash execute.sh applescript --preset open-file --path '/Users/me/doc.txt'
+# Type text (auto-switches to ABC input)
+bash execute.sh '{"action":"type","text":"Hello, world!"}'
 
-# Extract all text from an app
-bash execute.sh get-text --app 'Notes'
+# Key combo
+bash execute.sh '{"action":"key","key":"command+c"}'
+bash execute.sh '{"action":"key","key":"return"}'
 
-# Scroll in an app
-bash execute.sh scroll --app 'Safari' --direction down --amount 5
+# Scroll down at position
+bash execute.sh '{"action":"scroll","x":500,"y":400,"dy":-5}'
 
-# Focus (activate) an app
-bash execute.sh focus --app 'Terminal'
+# Drag from point A to point B
+bash execute.sh '{"action":"drag","x1":100,"y1":200,"x2":400,"y2":500}'
 
-# Check accessibility permission
-bash execute.sh check-access
+# Focus an app
+bash execute.sh '{"action":"focus","app":"Safari"}'
 
-# Connect to Chrome via CDP
-bash execute.sh chrome-connect --port 9222
+# Open URL (bypasses keyboard entirely)
+bash execute.sh '{"action":"open-url","url":"https://example.com","app":"Google Chrome"}'
 
-# Chrome: list tabs
-bash execute.sh chrome-tabs --port 9222
+# List running GUI apps
+bash execute.sh '{"action":"list-apps"}'
+```
 
-# Chrome: run JS in a tab
-bash execute.sh chrome-eval --port 9222 --tab 0 --code 'document.title'
+### find -- Locate UI elements by visual analysis
+```bash
+bash execute.sh '{"action":"find","mode":"button"}'
+bash execute.sh '{"action":"find","mode":"avatar"}'
+bash execute.sh '{"action":"find","mode":"color","target":"0,120,215"}'
 ```
+Returns screen coordinates of found elements. Use with click to interact.
 
-## Control Methods
-
-### Layer 1: App Discovery
-- `list-apps` — Lists all foreground apps with bundle IDs and available control methods
-- `app-info` — Gets detailed info: windows, UI elements, controllability
-
-### Layer 2: AppleScript/JXA (native macOS apps)
-- `applescript` — Run custom AppleScript or use presets
-- Presets: `activate-app`, `get-window-list`, `click-menu-item`, `open-file`, `get-clipboard`, `set-clipboard`
-- Works with: Finder, Terminal, Safari, Mail, Calendar, Notes, Sublime Text, etc.
-
-### Layer 3: Accessibility API (universal)
-- `ui-tree` — Read any app's UI element tree (roles, titles, values, positions)
-- `click` — Click elements by role:name or coordinates
-- `type` — Type text in the focused field
-- `get-text` — Extract all visible text from an app
-- `scroll` — Scroll within an app
-- `focus` — Bring an app to the foreground
-- `screenshot` — Capture app window or full screen
-- Requires: System Settings > Privacy & Security > Accessibility permission
-
-### Layer 4: Playwright CDP (Chrome web apps)
-- `chrome-connect` — Connect to Chrome via Chrome DevTools Protocol
-- `chrome-tabs` — List open Chrome tabs
-- `chrome-eval` — Run JavaScript in a Chrome tab
-- Requires: Chrome launched with `--remote-debugging-port=9222`
-
-## Safety Rules
-
-1. NEVER close or kill apps without explicit user permission
-2. NEVER interact with password fields or sensitive data
-3. Always screenshot BEFORE and AFTER any action for audit trail
-4. All actions logged to `~/.crewly/logs/computer-use.log`
-5. iPad apps: READ ONLY via Accessibility — never killall, never force-quit
+**Modes:**
+- **button** -- Finds bright/white rectangular regions (buttons, panels, input fields) by scanning for horizontal runs of light pixels (r,g,b > 200) wider than 100px, then clustering vertically adjacent runs into distinct elements.
+- **avatar** -- Finds colored circles in the center half of the screen (account avatars, profile icons) by detecting saturated pixels, then clusters them and offsets coordinates rightward to target adjacent text/labels.
+- **color** -- Finds regions matching a specific RGB color with tolerance of 60 per channel. Pass `target` as `"r,g,b"` (e.g. `"0,120,215"` for Windows blue). Returns the centroid and up to 5 sample points.
 
 ## Requirements
 
 - macOS (Apple Silicon or Intel)
 - `jq` for JSON parsing
-- Accessibility permission for UI reading/interaction
-- Chrome with `--remote-debugging-port` for CDP features
+- `sips`, `screencapture`, `osascript` (macOS built-in)
+- `python3` + `PIL`/`Pillow` (for grid overlay, crop, and find action)
+- Accessibility permission for click, type, key, scroll, drag, focus
+- Screen Recording permission for screenshot and find