ask-computer 0.1.0

data/lib/ask/skills/computer.use_computer/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: computer.use_computer
+description: Drive desktop apps in the background via Cua Driver and query encrypted Computer History — screenshots, clicks, typing, sandboxed VMs, and prior-run context
+---
+
+Use this skill when you need to automate desktop applications, inspect screen state, or resume prior work using Computer History.
+
+## Prerequisites
+
+Cua Driver must be installed. If `Ask::Computer.driver.start` fails, tell the user how to install it:
+
+```bash
+/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
+# then follow post-install instructions at https://cua.ai/docs/tutorials/drive-your-first-app
+```
+
+For Computer History (prior-run context), the daemon must be a nightly build with the preview admitted and history enabled:
+
+```bash
+cua-driver history enable
+cua-driver history status
+```
+
+All history access is permission-gated and metadata-only. Never assume history is available.
+
+## Step 1: Discover and Check History
+
+Before broader desktop inspection, check whether history is available — but only when the user asks to continue, resume, recall recent activity, or explain what a prior Cua run did. Do not query history for unrelated tasks.
+
+```ruby
+require "ask-computer"
+
+driver = Ask::Computer.driver
+driver.start
+
+history = Ask::Computer.history
+status = history.status
+
+# status fields: supported, admitted, enabled, paused, encrypted, profile,
+#                retention_days, quota_bytes, bytes_used, dropped_events, health
+# health values: ready, disabled, paused, not_admitted, key_locked, key_unavailable,
+#                storage_corrupt, quota_reached, events_dropped, writer_stopped
+
+if status.enabled? && status.healthy?
+  result = history.recent(limit: 20)
+  # result.events => Array<Ask::Computer::Event>  (CloudEvents 1.0, see references/history.md)
+  # result.metadata_only? => true  (always — no screenshots or typed text)
+else
+  # Continue without history. Preserve the reason (disabled/paused/unhealthy) in reasoning.
+end
+```
+
+If `history_status` or `history_query` is not advertised, or permission is denied, continue without history. Do not retry with a broader tool, read files directly, or ask the model to reconstruct history.
+
+## Step 2: Drive the Desktop
+
+Use the computer tools to observe and act. Always verify state changes with a screenshot after acting.
+
+```ruby
+# Observe
+sandbox = Ask::Computer.sandbox
+image = sandbox.screenshot  # or driver.call_tool("computer_screenshot", {})
+
+# Act (background — does not steal cursor or focus)
+driver.call_tool("computer_click", { x: 120, y: 80 })
+driver.call_tool("computer_type", { text: "Hello" })
+driver.call_tool("computer_key", { key: "Enter" })
+
+# Or via Ask::Tool subclasses (for agent sessions)
+# Ask::Tools::Computer::Screenshot, Click, Type, Key, HistoryStatus, HistoryQuery
+```
+
+Prefer accessibility-aware targeting when available over raw coordinates. After each state-changing action, re-screenshot to confirm the change.
+
+## Step 3: Query History with Bounded Reads
+
+Events are ordered by `data.sequence` ascending. Treat missing sequence numbers as gap evidence. Paginate toward older records with `until_sequence`; toward newer with `since_sequence`. Bounds are inclusive — subtract/add 1 for non-overlapping pages.
+
+```ruby
+# Filtered query
+history.query(limit: 20, session_id: "abc", since_sequence: 40, until_sequence: 100)
+
+# Paginate through all recent history (newest-first, bounded pages)
+history.each_page(limit: 50) do |page|
+  page.events.each { |e| puts "#{e.type} seq=#{e.data[:sequence]}" }
+end
+
+# Ergonomic: maps RFC error codes to typed errors
+#   HistoryNotAdmitted, HistoryKeyLocked, HistoryStorageCorrupt, InvalidHistoryQuery, etc.
+```
+
+Handle health warnings explicitly:
+
+- `dropped_events > 0` or `health == "events_dropped"` → treat interval as incomplete.
+- `health == "quota_reached"` → history after that point is incomplete.
+- `health == "storage_corrupt"` → stop querying; direct user to `cua-driver history delete --yes` if they accept data loss.
+- `health == "key_locked"` → user must unlock Keychain/Credential Manager; do not prompt via another tool.
+
+## Step 4: Use History as a Lead, Not a Transcript
+
+Returned events contain only: time, sequence, session/action IDs, capability, optional `application` (`bundle_id`, `display_name`), outcome/route/delivery/evidence categories. They never contain screenshots, typed text, clipboard, tool args/results, window titles, or URLs.
+
+Treat events as hints. Use an identified `application` or `capability` to locate the app, then verify live state (screenshot, accessibility tree) before acting. Never treat history as a complete transcript.
+
+## Sandbox VMs (Optional)
+
+For isolated or parallel runs, use sandboxed VMs:
+
+```ruby
+sandbox = Ask::Computer.sandbox
+
+sandbox.ephemeral(:linux) do |vm|
+  vm.shell("echo hello")
+  vm.screenshot
+  vm.click(100, 200)
+  vm.type("Hello from sandbox")
+end
+# vm.destroy is called automatically on block exit
+```
+
+Requires a driver that advertises `sandbox_create` / `sandbox_*` tools. Degrade gracefully when unavailable.

data/lib/ask/skills/computer.use_computer/references/history.md

@@ -0,0 +1,80 @@
+# Computer History Reference
+
+Detailed schema for `Ask::Computer::History` responses. The live contract is the Cua RFC; this file is a local summary so the skill stays compact.
+
+## Status Response
+
+```
+{
+  supported: bool, admitted: bool, enabled: bool, paused: bool,
+  encrypted: bool, profile: String, retention_days: int (default 7),
+  quota_bytes: int (default 104857600), bytes_used: int,
+  dropped_events: int, health: String
+}
+```
+
+Health values:
+
+| health | meaning |
+|---|---|
+| ready | Healthy |
+| disabled | Capture off |
+| paused | Capture paused, history still queryable |
+| not_admitted | Daemon not started with preview flag |
+| key_unavailable / key_locked / key_corrupt / key_destroy_failed | Credential store issue |
+| storage_unavailable / storage_corrupt | Encrypted store unreadable |
+| quota_reached | Store at quota, new events may be incomplete |
+| events_dropped | Nonblocking writer dropped events |
+| writer_stopped | Writer unavailable |
+
+## Event (CloudEvents 1.0, dataschema urn:cua-driver:schema:history-event:v0)
+
+Each `history_query` event:
+
+```
+{
+  specversion, id, source, type, subject, time, datacontenttype, dataschema,
+  data: {
+    session_id, action_id, sequence, platform, process_model, capability,
+    caller_category,
+    application?: { bundle_id?, display_name? },
+    payload: { kind, effect?, route?, delivery?, delivered_count?, evidence_kinds?[] }
+  }
+}
+```
+
+### Event Types
+
+| type | payload kind |
+|---|---|
+| cua-driver.history.control.v0 | control |
+| cua-driver.history.action_started.v0 | action_started |
+| cua-driver.history.action_completed.v0 | action_completed |
+| cua-driver.history.session_started.v0 | session |
+| cua-driver.history.session_ended.v0 | session |
+| cua-driver.history.access.v0 | access |
+| cua-driver.history.health.v0 | health |
+
+Ordering: `data.sequence` ascending. `limit` keeps newest N then returns them ascending. No pagination token — page with `since_sequence` / `until_sequence` (inclusive).
+
+## Error Codes
+
+Tool failures map to typed errors via `Ask::Computer::Error.from_code`:
+
+- `invalid_history_query` / `invalid_history_query_range` → `InvalidHistoryQuery*`
+- `history_preview_not_admitted` → `HistoryNotAdmitted`
+- `history_key_*` → `HistoryKey*`
+- `history_storage_*` → `HistoryStorage*`
+- `history_quota_reached` → `HistoryQuotaReached`
+- `history_writer_stopped` → `HistoryWriterStopped`
+- `history_events_dropped` → `HistoryEventsDropped`
+
+Permission denials surface as `Ask::Computer::PermissionDenied` (do not retry with a broader tool).
+
+## Storage Profile
+
+CBOR Sequence of COSE_Encrypt0, CloudEvents JSON inside, ChaCha20-Poly1305, HKDF per chunk, root key in platform credential store (Keychain / Credential Manager / Secret Service). No plaintext fallback.
+
+For full spec see:
+- https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-preview.md
+- https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-agent-integration-rfc.md

data/lib/ask/tools/computer/click.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class Click < Ask::Tool
+        description "Click at screen coordinates via Cua Driver. Background click — does not steal cursor."
+
+        param :x, type: :integer, desc: "X coordinate in screen pixels", required: true
+        param :y, type: :integer, desc: "Y coordinate in screen pixels", required: true
+        param :button, type: :string, desc: "Mouse button: left, right, middle", required: false
+
+        def execute(x:, y:, button: "left")
+          driver = Ask::Computer.driver
+          result = driver.call_tool("computer_click", { x: Integer(x), y: Integer(y), button: button.to_s })
+          Ask::Result.ok(data: result)
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        rescue ArgumentError => e
+          Ask::Result.error(message: e.message)
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/history_query.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class HistoryQuery < Ask::Tool
+        description "Query encrypted Computer History for a bounded slice of metadata-only events. " \
+                    "Returns at most 200 events ordered by sequence. Requires history.query capability. " \
+                    "Check history_status first. Results are metadata-only — no screenshots or typed text."
+
+        param :limit, type: :integer, desc: "Max events to return (1..200, default 50)", required: false
+        param :session_id, type: :string, desc: "Opaque session ID filter (1..128 chars)", required: false
+        param :since_sequence, type: :integer, desc: "Inclusive lower sequence bound (>=1)", required: false
+        param :until_sequence, type: :integer, desc: "Inclusive upper sequence bound (>=1)", required: false
+
+        def execute(limit: 50, session_id: nil, since_sequence: nil, until_sequence: nil)
+          history = Ask::Computer.history
+          result = history.query(
+            limit: limit,
+            session_id: session_id,
+            since_sequence: since_sequence,
+            until_sequence: until_sequence
+          )
+
+          Ask::Result.ok(data: {
+            events: result.events.map { |e| serialize_event(e) },
+            metadata_only: result.metadata_only,
+            model_context_disclosure: result.model_context_disclosure
+          })
+        rescue Ask::Computer::InvalidHistoryQuery, Ask::Computer::InvalidHistoryQueryRange => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name, code: e.class.name })
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+
+        private
+
+        def serialize_event(event)
+          {
+            specversion: event.specversion,
+            id: event.id,
+            source: event.source,
+            type: event.type,
+            subject: event.subject,
+            time: event.time,
+            datacontenttype: event.datacontenttype,
+            dataschema: event.dataschema,
+            data: event.data
+          }
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/history_status.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class HistoryStatus < Ask::Tool
+        description "Check Cua Computer History status — whether capture is enabled, " \
+                    "paused, healthy, and how much encrypted storage is used. " \
+                    "Call this before history_query."
+
+        def execute
+          history = Ask::Computer.history
+          status = history.status
+
+          Ask::Result.ok(data: {
+            supported: status.supported,
+            admitted: status.admitted,
+            enabled: status.enabled,
+            paused: status.paused,
+            encrypted: status.encrypted,
+            profile: status.profile,
+            retention_days: status.retention_days,
+            quota_bytes: status.quota_bytes,
+            bytes_used: status.bytes_used,
+            dropped_events: status.dropped_events,
+            health: status.health
+          })
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/installer_setup.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class InstallerSetup < Ask::Tool
+        description "Install Cua Driver (if missing) and optionally enable Computer History. " \
+                    "Use dry_run: true to preview the shell command without executing it. " \
+                    "Channel is stable or nightly."
+
+        param :channel, type: :string, desc: "stable or nightly", required: false
+        param :enable_history, type: :boolean, desc: "Enable Computer History after install", required: false
+        param :dry_run, type: :boolean, desc: "Preview only, do not run installer", required: false
+
+        def execute(channel: "stable", enable_history: false, dry_run: false)
+          steps = Ask::Computer::Installer.setup(
+            channel: channel,
+            enable_history: enable_history,
+            dry_run: dry_run
+          )
+          Ask::Result.ok(data: {
+            steps: steps.map { |name, r| { name: name.to_s, ok: r.ok?, message: r.message, command: r.command } }
+          })
+        rescue StandardError => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/installer_status.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class InstallerStatus < Ask::Tool
+        description "Check whether Cua Driver is installed and report its version and " \
+                    "Computer History health. Read-only, safe to call without the daemon."
+
+        def execute
+          s = Ask::Computer::Installer.status
+          Ask::Result.ok(data: {
+            installed: s[:installed],
+            version: s[:version],
+            bin: s[:bin],
+            history: s[:history]
+          })
+        rescue StandardError => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/key.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class Key < Ask::Tool
+        description "Press a key or key combination via Cua Driver (e.g. Enter, Tab, Control+C)."
+
+        param :key, type: :string, desc: "Key or combination to press", required: true
+
+        def execute(key:)
+          driver = Ask::Computer.driver
+          result = driver.call_tool("computer_key", { key: key.to_s })
+          Ask::Result.ok(data: result)
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/screenshot.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class Screenshot < Ask::Tool
+        description "Capture a screenshot of the current desktop via Cua Driver. " \
+                    "Returns an image that vision models can analyze."
+
+        def execute
+          sandbox = Ask::Computer.sandbox
+          image = sandbox.screenshot
+          Ask::Result.ok(data: image)
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/computer/type.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "ask/tools"
+
+module Ask
+  module Tools
+    module Computer
+      class Type < Ask::Tool
+        description "Type text via Cua Driver. Sends keystrokes in the background."
+
+        param :text, type: :string, desc: "Text to type", required: true
+
+        def execute(text:)
+          driver = Ask::Computer.driver
+          result = driver.call_tool("computer_type", { text: text.to_s })
+          Ask::Result.ok(data: result)
+        rescue Ask::Computer::Error => e
+          Ask::Result.error(message: e.message, metadata: { error_class: e.class.name })
+        end
+      end
+    end
+  end
+end

data/lib/ask-computer.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "ask"
+require "ask/tools"
+require_relative "ask/computer/version"
+require_relative "ask/computer/errors"
+require_relative "ask/computer/config"
+require_relative "ask/computer/context"
+require_relative "ask/computer"
+require_relative "ask/computer/driver"
+require_relative "ask/computer/app"
+require_relative "ask/computer/history"
+require_relative "ask/computer/sandbox"
+require_relative "ask/computer/installer"
+require_relative "ask/computer/cli"
+require_relative "ask/tools/computer/history_status"
+require_relative "ask/tools/computer/history_query"
+require_relative "ask/tools/computer/screenshot"
+require_relative "ask/tools/computer/click"
+require_relative "ask/tools/computer/type"
+require_relative "ask/tools/computer/key"
+require_relative "ask/tools/computer/installer_status"
+require_relative "ask/tools/computer/installer_setup"

metadata

@@ -0,0 +1,156 @@
+--- !ruby/object:Gem::Specification
+name: ask-computer
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Kaka Ruto
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: ask-core
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.11.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.11.3
+- !ruby/object:Gem::Dependency
+  name: ask-mcp
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.4.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.4.0
+- !ruby/object:Gem::Dependency
+  name: ask-tools
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.6.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.6.2
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.25'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.25'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+description: Drive desktop apps in the background via Cua Driver — screenshots, clicks,
+  typing, sandboxed VMs, and encrypted Computer History. Wraps Cua's MCP tools as
+  Ask::Tool subclasses and ships a bundled computer.use_computer skill.
+email:
+- kaka@myrrlabs.com
+executables:
+- ask-computer
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- exe/ask-computer
+- lib/ask-computer.rb
+- lib/ask/computer.rb
+- lib/ask/computer/app.rb
+- lib/ask/computer/cli.rb
+- lib/ask/computer/config.rb
+- lib/ask/computer/context.rb
+- lib/ask/computer/driver.rb
+- lib/ask/computer/errors.rb
+- lib/ask/computer/history.rb
+- lib/ask/computer/installer.rb
+- lib/ask/computer/sandbox.rb
+- lib/ask/computer/version.rb
+- lib/ask/skills/computer.use_computer/SKILL.md
+- lib/ask/skills/computer.use_computer/references/history.md
+- lib/ask/tools/computer/click.rb
+- lib/ask/tools/computer/history_query.rb
+- lib/ask/tools/computer/history_status.rb
+- lib/ask/tools/computer/installer_setup.rb
+- lib/ask/tools/computer/installer_status.rb
+- lib/ask/tools/computer/key.rb
+- lib/ask/tools/computer/screenshot.rb
+- lib/ask/tools/computer/type.rb
+homepage: https://github.com/ask-rb/ask-computer
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/ask-rb/ask-computer
+  source_code_uri: https://github.com/ask-rb/ask-computer
+  changelog_uri: https://github.com/ask-rb/ask-computer/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.18
+specification_version: 4
+summary: Computer use for the ask-rb ecosystem
+test_files: []