screenhand 0.4.1 → 0.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +18 -2
  2. package/dist-app-maps/com.apple.Music.json +6587 -0
  3. package/dist-app-maps/com.apple.Notes.json +6098 -0
  4. package/dist-app-maps/com.apple.Photos.json +406 -0
  5. package/dist-app-maps/com.apple.Terminal.json +6457 -0
  6. package/dist-app-maps/com.apple.finder.json +420 -0
  7. package/dist-app-maps/com.apple.iCal.json +2375 -0
  8. package/dist-app-maps/com.apple.iWork.Keynote.json +2374 -0
  9. package/dist-app-maps/com.apple.iWork.Pages.json +7564 -0
  10. package/dist-app-maps/com.apple.mail.json +8353 -0
  11. package/dist-app-maps/com.apple.reminders.json +3322 -0
  12. package/dist-app-maps/net.whatsapp.WhatsApp.json +5151 -0
  13. package/dist-playbooks/calendar-create-event.json +20 -0
  14. package/dist-playbooks/calendar-list-events.json +20 -0
  15. package/dist-playbooks/calendar-navigate-views.json +47 -0
  16. package/dist-playbooks/calendar-open-settings.json +20 -0
  17. package/dist-playbooks/google-ads-transparency-competitor-research.json +89 -0
  18. package/dist-playbooks/google-search-competitor-research.json +76 -0
  19. package/dist-playbooks/keynote-add-slide.json +20 -0
  20. package/dist-playbooks/keynote-create-presentation.json +20 -0
  21. package/dist-playbooks/keynote-export-pdf.json +20 -0
  22. package/dist-playbooks/keynote-play-slideshow.json +20 -0
  23. package/dist-playbooks/meta-ad-library-competitor-research.json +100 -0
  24. package/dist-playbooks/notes-mastery-workflows.json +468 -0
  25. package/dist-playbooks/pages-export-pdf.json +20 -0
  26. package/dist-playbooks/pages-new-document.json +20 -0
  27. package/dist-playbooks/pages-open-document.json +20 -0
  28. package/dist-playbooks/reminders-complete.json +21 -0
  29. package/dist-playbooks/reminders-create.json +21 -0
  30. package/dist-playbooks/reminders-list.json +22 -0
  31. package/dist-playbooks/reminders-open.json +35 -0
  32. package/dist-playbooks/whatsapp-contact-info.json +32 -0
  33. package/dist-playbooks/whatsapp-navigate.json +71 -0
  34. package/dist-playbooks/whatsapp-new-call.json +32 -0
  35. package/dist-playbooks/whatsapp-new-group.json +32 -0
  36. package/dist-playbooks/whatsapp-search.json +28 -0
  37. package/dist-playbooks/whatsapp-settings.json +23 -0
  38. package/dist-playbooks/x_change_avatar.json +52 -0
  39. package/dist-references/apple-music.json +822 -0
  40. package/dist-references/calendar.json +1020 -0
  41. package/dist-references/google-search-competitor-research.json +73 -0
  42. package/dist-references/keynote.json +134 -0
  43. package/dist-references/mail.json +431 -0
  44. package/dist-references/pages.json +1203 -0
  45. package/dist-references/photos.json +642 -0
  46. package/dist-references/reminders.json +835 -0
  47. package/dist-references/terminal.json +640 -0
  48. package/dist-references/whatsapp.json +324 -0
  49. package/package.json +1 -1
package/README.md CHANGED
@@ -111,7 +111,7 @@ Launch Chrome with remote debugging to enable browser tools:
111
111
  open -a "Google Chrome" --args --remote-debugging-port=9222
112
112
  ```
113
113
 
114
- That's it. Your AI client now has 111 tools for desktop automation.
114
+ That's it. Your AI client now has 111 tools for desktop automation — and ships with prebuilt knowledge for 36 apps so you don't start from zero.
115
115
 
116
116
  <details>
117
117
  <summary><b>Building from source</b> (contributors only)</summary>
@@ -126,6 +126,20 @@ On Windows, use `npm run build:native:windows` instead.
126
126
 
127
127
  ---
128
128
 
129
+ ## Prebuilt Platform Knowledge
130
+
131
+ Every install ships with battle-tested knowledge so AI starts from EXPERT level on day one — no re-exploration needed:
132
+
133
+ | | Count | Apps Included |
134
+ |---|---|---|
135
+ | **References** | 36 | Terminal, Mail, Finder, Calendar, Reminders, Keynote, Pages, Notes, Photos, Apple Music, WhatsApp, Figma, Discord, DaVinci Resolve, Canva, Instagram, X/Twitter, LinkedIn, YouTube, Reddit, Notion, n8n, and more |
136
+ | **Playbooks** | 49 | Calendar events, Keynote decks, Reminders, Notes workflows, WhatsApp navigation, DaVinci color grading/render, Canva carousel, social posting, Google Flow, competitor research, and more |
137
+ | **App Maps** | 14 | Spatial UI blueprints for Finder, Mail, Calendar, Notes, Reminders, Keynote, Pages, Photos, Apple Music, Terminal, WhatsApp, Figma, Discord, Notion |
138
+
139
+ These load automatically when the matching app or website is detected. No setup required.
140
+
141
+ ---
142
+
129
143
  ## What It Does
130
144
 
131
145
  ScreenHand gives AI agents eight capabilities:
@@ -272,6 +286,8 @@ Computer Use is cloud-based and screenshot-driven. ScreenHand is local-first, us
272
286
  <summary><b>What apps can it control?</b></summary>
273
287
 
274
288
  Any app with Accessibility support (most macOS/Windows apps). Chrome and Electron apps get full DOM access via CDP. Canvas-heavy apps (games, Photoshop viewport) use OCR as fallback.
289
+
290
+ Ships with EXPERT-level prebuilt knowledge for: Terminal, Mail, Finder, Calendar, Reminders, Keynote, Pages, Notes, Photos, Apple Music, WhatsApp, Figma, Discord, DaVinci Resolve, Canva, Instagram, X/Twitter, LinkedIn, YouTube, Reddit, Notion, n8n, and more. Any other app gets explored and learned automatically on first use.
275
291
  </details>
276
292
 
277
293
  <details>
@@ -297,7 +313,7 @@ Accessibility: ~50ms. Chrome CDP: ~10ms (background, no focus needed). OCR: ~600
297
313
  ```bash
298
314
  git clone https://github.com/manushi4/screenhand.git
299
315
  cd screenhand && npm install && npm run build:native
300
- npm test # 1405 tests, 56 files
316
+ npm test # 1331 tests, 54 files
301
317
  ```
302
318
 
303
319
  ## Contact