@soederpop/luca 0.0.29 → 0.0.30

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. package/commands/try-all-challenges.ts +1 -1
  2. package/docs/TABLE-OF-CONTENTS.md +0 -3
  3. package/docs/tutorials/20-browser-esm.md +234 -0
  4. package/package.json +1 -1
  5. package/src/agi/container.server.ts +4 -0
  6. package/src/agi/features/assistant.ts +62 -1
  7. package/src/agi/features/browser-use.ts +623 -0
  8. package/src/bootstrap/generated.ts +236 -308
  9. package/src/cli/build-info.ts +2 -2
  10. package/src/clients/rest.ts +7 -7
  11. package/src/commands/chat.ts +22 -0
  12. package/src/commands/describe.ts +67 -2
  13. package/src/commands/prompt.ts +23 -3
  14. package/src/container.ts +411 -113
  15. package/src/helper.ts +189 -5
  16. package/src/introspection/generated.agi.ts +17148 -11148
  17. package/src/introspection/generated.node.ts +5179 -2200
  18. package/src/introspection/generated.web.ts +379 -291
  19. package/src/introspection/index.ts +7 -0
  20. package/src/introspection/scan.ts +224 -7
  21. package/src/node/container.ts +31 -10
  22. package/src/node/features/content-db.ts +7 -7
  23. package/src/node/features/disk-cache.ts +11 -11
  24. package/src/node/features/esbuild.ts +3 -3
  25. package/src/node/features/file-manager.ts +15 -15
  26. package/src/node/features/fs.ts +23 -22
  27. package/src/node/features/git.ts +10 -10
  28. package/src/node/features/ink.ts +13 -13
  29. package/src/node/features/ipc-socket.ts +8 -8
  30. package/src/node/features/networking.ts +3 -3
  31. package/src/node/features/os.ts +7 -7
  32. package/src/node/features/package-finder.ts +15 -15
  33. package/src/node/features/proc.ts +1 -1
  34. package/src/node/features/ui.ts +13 -13
  35. package/src/node/features/vm.ts +4 -4
  36. package/src/scaffolds/generated.ts +1 -1
  37. package/src/servers/express.ts +6 -6
  38. package/src/servers/mcp.ts +4 -4
  39. package/src/servers/socket.ts +6 -6
  40. package/docs/apis/features/node/window-manager.md +0 -445
  41. package/docs/examples/window-manager-layouts.md +0 -180
  42. package/docs/examples/window-manager.md +0 -125
  43. package/docs/window-manager-fix.md +0 -249
  44. package/scripts/test-window-manager-lifecycle.ts +0 -86
  45. package/scripts/test-window-manager.ts +0 -43
  46. package/src/node/features/window-manager.ts +0 -1603
@@ -1,125 +0,0 @@
1
- ---
2
- title: "Window Manager"
3
- tags: [windowManager, native, ipc, macos, browser, window]
4
- lastTested: null
5
- lastTestPassed: null
6
- ---
7
-
8
- # windowManager
9
-
10
- Native window control via LucaVoiceLauncher IPC. Communicates with the macOS launcher app over a Unix domain socket using NDJSON to spawn, navigate, screenshot, and manage native browser windows.
11
-
12
- ## Overview
13
-
14
- Use the `windowManager` feature when you need to control native browser windows from Luca. It acts as an IPC server that the LucaVoiceLauncher macOS app connects to. Through this connection you can spawn windows with configurable chrome, navigate URLs, evaluate JavaScript, capture screenshots, and record video.
15
-
16
- Requires the LucaVoiceLauncher native macOS app to be running and connected.
17
-
18
- ## Enabling the Feature
19
-
20
- ```ts
21
- const wm = container.feature('windowManager', {
22
- autoListen: false,
23
- requestTimeoutMs: 10000
24
- })
25
- console.log('Window Manager feature created')
26
- console.log('Listening:', wm.isListening)
27
- console.log('Client connected:', wm.isClientConnected)
28
- ```
29
-
30
- ## API Documentation
31
-
32
- ```ts
33
- const info = await container.features.describe('windowManager')
34
- console.log(info)
35
- ```
36
-
37
- ## Spawning Windows
38
-
39
- Create native browser windows with configurable dimensions and chrome.
40
-
41
- ```ts skip
42
- const result = await wm.spawn({
43
- url: 'https://google.com',
44
- width: 1024,
45
- height: 768,
46
- alwaysOnTop: true,
47
- window: { decorations: 'hiddenTitleBar', shadow: true }
48
- })
49
- console.log('Window ID:', result.windowId)
50
- ```
51
-
52
- The `spawn()` method sends a dispatch to the native app and waits for acknowledgement. Window options include position, size, transparency, click-through, and title bar style.
53
-
54
- ## Navigation and JavaScript Evaluation
55
-
56
- Control window content after spawning.
57
-
58
- ```ts skip
59
- const handle = wm.window(result.windowId)
60
- await handle.navigate('https://news.ycombinator.com')
61
- console.log('Navigated')
62
-
63
- const title = await handle.eval('document.title')
64
- console.log('Page title:', title)
65
-
66
- await handle.focus()
67
- await handle.close()
68
- ```
69
-
70
- The `window()` method returns a `WindowHandle` for chainable operations on a specific window. Use `eval()` to run JavaScript in the window's web view.
71
-
72
- ## Screenshots and Video
73
-
74
- Capture visual output from windows.
75
-
76
- ```ts skip
77
- await wm.screengrab({
78
- windowId: result.windowId,
79
- path: './screenshot.png'
80
- })
81
- console.log('Screenshot saved')
82
-
83
- await wm.video({
84
- windowId: result.windowId,
85
- path: './recording.mp4',
86
- durationMs: 5000
87
- })
88
- console.log('Video recorded')
89
- ```
90
-
91
- Screenshots are saved as PNG. Video recording captures for the specified duration.
92
-
93
- ## Terminal Windows
94
-
95
- Spawn native terminal windows that render command output with ANSI support.
96
-
97
- ```ts skip
98
- const tty = await wm.spawnTTY({
99
- command: 'htop',
100
- title: 'System Monitor',
101
- width: 900,
102
- height: 600,
103
- cols: 120,
104
- rows: 40
105
- })
106
- console.log('TTY window:', tty.windowId)
107
- ```
108
-
109
- Terminal windows are read-only displays of process output. Closing the window terminates the process.
110
-
111
- ## IPC Communication
112
-
113
- Other features can send arbitrary messages over the socket connection.
114
-
115
- ```ts skip
116
- wm.listen()
117
- wm.on('message', (msg) => console.log('App says:', msg))
118
- wm.send({ id: 'abc', status: 'ready', speech: 'Window manager online' })
119
- ```
120
-
121
- The `message` event fires for any non-windowAck message from the native app.
122
-
123
- ## Summary
124
-
125
- The `windowManager` feature provides native window control through IPC with the LucaVoiceLauncher app. Spawn browser windows, navigate, evaluate JS, capture screenshots, and record video. Supports terminal windows for command output. Key methods: `spawn()`, `navigate()`, `eval()`, `screengrab()`, `video()`, `spawnTTY()`, `window()`.
@@ -1,249 +0,0 @@
1
- # Window Manager Fix
2
-
3
- ## Problem
4
-
5
- The current `windowManager` design allows any Luca process to call `listen()` on the same well-known Unix socket:
6
-
7
- - `~/Library/Application Support/LucaVoiceLauncher/ipc-window.sock`
8
-
9
- That means unrelated commands can compete for ownership of the app-facing socket. The current implementation makes this worse by doing the following on startup:
10
-
11
- 1. If the socket path exists, `unlinkSync(socketPath)`.
12
- 2. Bind a new server at the same path.
13
-
14
- This creates a race where one Luca process can steal the socket from another. The native `LucaVoiceLauncher` app then disconnects from the old server and reconnects to whichever process currently owns the path. If that process exits, the app falls into reconnect loops.
15
-
16
- This is the root cause of the observed behavior where:
17
-
18
- - the launcher sometimes connects successfully
19
- - the connection then drops unexpectedly
20
- - repeated `ipc connect failed` messages appear in the launcher log
21
-
22
- ## Design Goal
23
-
24
- We want:
25
-
26
- - one stable owner of the app-facing socket
27
- - many independent Luca commands able to trigger window actions
28
- - optional failover if the main owner dies
29
- - support for multiple launcher app clients over time, and optionally at once
30
-
31
- The key design rule is:
32
-
33
- > Many clients is fine. Many servers competing for the same well-known socket is not.
34
-
35
- ## Recommended Architecture
36
-
37
- ### 1. Single broker for the app socket
38
-
39
- Only one broker process may own:
40
-
41
- - `ipc-window.sock`
42
-
43
- The broker is responsible for:
44
-
45
- - accepting native launcher app connections
46
- - tracking connected app clients
47
- - routing window commands to the selected app client
48
- - receiving `windowAck`, `windowClosed`, and `terminalExited`
49
- - routing responses and lifecycle events back to the original requester
50
-
51
- ### 2. Separate control channel for Luca commands
52
-
53
- Luca commands should not bind the app-facing socket directly.
54
-
55
- Instead, they should talk to the broker over a separate channel, for example:
56
-
57
- - `~/Library/Application Support/LucaVoiceLauncher/ipc-window-control.sock`
58
-
59
- This control channel is for producers:
60
-
61
- - `luca main`
62
- - `luca workflow run ...`
63
- - `luca present`
64
- - scripts
65
- - background jobs
66
-
67
- These producers send requests to the broker, and the broker fans them out to the connected app client.
68
-
69
- ### 3. Broker supports multiple app clients
70
-
71
- The broker should replace the current single `_client` field with a registry:
72
-
73
- ```ts
74
- Map<string, ClientConnection>
75
- ```
76
-
77
- Each client should have:
78
-
79
- - `clientId`
80
- - `socket`
81
- - `buffer`
82
- - metadata if useful later, such as display, role, labels, or lastSeenAt
83
-
84
- This allows:
85
-
86
- - multiple launcher app instances
87
- - reconnect without confusing request ownership
88
- - future routing by target client
89
-
90
- ## Routing Model
91
-
92
- ### Producer -> broker
93
-
94
- Producer sends a request like:
95
-
96
- ```json
97
- {
98
- "type": "windowRequest",
99
- "requestId": "uuid",
100
- "originId": "uuid",
101
- "targetClientId": "optional",
102
- "window": {
103
- "action": "open",
104
- "url": "https://example.com"
105
- }
106
- }
107
- ```
108
-
109
- ### Broker -> app client
110
-
111
- Broker forwards the request to the chosen app client, preserving `requestId`.
112
-
113
- ### App client -> broker
114
-
115
- App replies with:
116
-
117
- - `windowAck`
118
- - `windowClosed`
119
- - `terminalExited`
120
-
121
- ### Broker -> producer
122
-
123
- Broker routes:
124
-
125
- - the `windowAck` back to the producer that originated the request
126
- - lifecycle events either to the originating producer, or to any subscribed producer
127
-
128
- ## Client Selection Policy
129
-
130
- The simplest policy is:
131
-
132
- - use the most recently connected healthy app client
133
-
134
- Later policies can support:
135
-
136
- - explicit `targetClientId`
137
- - labels like `role=presenter`
138
- - display-aware routing
139
- - sticky routing based on `windowId -> clientId`
140
-
141
- ## Leader Election / Failover
142
-
143
- If we want multiple `windowManager` instances to exist, they must not all behave as brokers.
144
-
145
- Instead:
146
-
147
- 1. Try connecting to the broker control socket.
148
- 2. If broker exists, act as a producer client.
149
- 3. If broker does not exist, try to acquire a broker lock.
150
- 4. If lock succeeds, become broker and bind both sockets.
151
- 5. If lock fails, retry broker connection and act as producer.
152
-
153
- Possible lock mechanisms:
154
-
155
- - lock file with `flock`
156
- - lock directory with atomic `mkdir`
157
- - local TCP/Unix registration endpoint
158
-
159
- The important constraint is:
160
-
161
- - only the elected broker binds `ipc-window.sock`
162
-
163
- All other instances must route through it.
164
-
165
- ## Why not let many processes bind the same socket?
166
-
167
- Because Unix domain socket paths are singular ownership points. A path is not a shared bus.
168
-
169
- If multiple processes all call `listen()` against the same path and delete stale files optimistically, they will:
170
-
171
- - steal the path from each other
172
- - disconnect the app unexpectedly
173
- - lose in-flight requests
174
- - create non-deterministic routing
175
-
176
- This is fundamentally the wrong abstraction.
177
-
178
- ## Backward-Compatible Migration
179
-
180
- We can migrate without breaking the public `windowManager.spawn()` API.
181
-
182
- ### Phase 1
183
-
184
- - Introduce a broker mode internally.
185
- - Add `ipc-window-control.sock`.
186
- - Keep the existing app protocol unchanged.
187
- - Make `windowManager.spawn()` talk to the broker when possible.
188
-
189
- ### Phase 2
190
-
191
- - Prevent non-broker processes from binding `ipc-window.sock`.
192
- - Replace blind `unlinkSync(socketPath)` with active listener detection.
193
- - Add broker election and failover.
194
-
195
- ### Phase 3
196
-
197
- - Add multi-client routing.
198
- - Add subscriptions for lifecycle events.
199
- - Add explicit target selection if needed.
200
-
201
- ## Minimal Fix if We Need Something Fast
202
-
203
- If we do not implement the full broker immediately, we should at least stop destroying active listeners.
204
-
205
- `listen()` should:
206
-
207
- 1. Attempt to connect to the existing socket.
208
- 2. If a listener is alive, do not unlink or rebind.
209
- 3. If the socket is dead, clean it up and bind.
210
-
211
- This does not solve multi-producer routing, but it prevents random Luca commands from stealing the app socket from a healthy broker.
212
-
213
- ## Proposed Internal Refactor
214
-
215
- Current state:
216
-
217
- - one process tries to be both broker and producer
218
- - one `_client`
219
- - one app-facing socket
220
-
221
- Target state:
222
-
223
- - broker owns app-facing socket
224
- - producers use control socket
225
- - broker stores:
226
- - `clients: Map<clientId, ClientConnection>`
227
- - `pendingRequests: Map<requestId, PendingRequest>`
228
- - `requestOrigins: Map<requestId, originConnection>`
229
- - `windowOwners: Map<windowId, clientId>`
230
-
231
- That separation gives us:
232
-
233
- - stable app connectivity
234
- - multi-command triggering
235
- - failover
236
- - room for multi-client routing
237
-
238
- ## Summary
239
-
240
- The right fix is not “allow many `listen()` calls on the same socket.”
241
-
242
- The right fix is:
243
-
244
- - one elected broker owns the app socket
245
- - many Luca processes talk to the broker
246
- - many app clients may connect to the broker
247
- - failover is implemented through broker election, not socket contention
248
-
249
- That preserves a stable connection for the launcher app while still allowing multiple people, commands, or workflows to trigger window operations.
@@ -1,86 +0,0 @@
1
- import { NodeContainer } from '../src/node/container'
2
-
3
- const args = new Set(process.argv.slice(2))
4
- const socketArg = process.argv.slice(2).find((arg) => arg.startsWith('--socket='))
5
- const socketPath = socketArg?.slice('--socket='.length)
6
- const shouldSpawnTTY = args.has('--spawn-tty')
7
- const shouldSpawnWindow = args.has('--spawn-window')
8
-
9
- const container = new NodeContainer({ cwd: process.cwd() })
10
- const wm = socketPath
11
- ? container.feature('windowManager', { socketPath })
12
- : container.feature('windowManager')
13
-
14
- wm.listen()
15
-
16
- const logCount = () => {
17
- console.log(`[state] windowCount=${wm.state.get('windowCount')}`)
18
- }
19
-
20
- wm.on('clientConnected', () => {
21
- console.log('[clientConnected] native launcher connected')
22
- })
23
-
24
- wm.on('clientDisconnected', () => {
25
- console.log('[clientDisconnected] native launcher disconnected')
26
- })
27
-
28
- wm.on('windowAck', (msg) => {
29
- console.log('[windowAck]', JSON.stringify(msg))
30
- logCount()
31
- })
32
-
33
- wm.on('windowClosed', (msg) => {
34
- console.log('[windowClosed]', JSON.stringify(msg))
35
- logCount()
36
- })
37
-
38
- wm.on('terminalExited', (msg) => {
39
- console.log('[terminalExited]', JSON.stringify(msg))
40
- })
41
-
42
- wm.on('message', (msg) => {
43
- if (msg?.type === 'windowAck' || msg?.type === 'windowClosed' || msg?.type === 'terminalExited') {
44
- return
45
- }
46
- console.log('[message]', JSON.stringify(msg))
47
- })
48
-
49
- let spawned = false
50
- wm.on('clientConnected', async () => {
51
- if (spawned) return
52
- spawned = true
53
-
54
- if (shouldSpawnWindow) {
55
- const opened = await wm.spawn({
56
- url: 'https://example.com',
57
- width: 900,
58
- height: 620,
59
- })
60
- console.log('[spawn-window]', opened)
61
- }
62
-
63
- if (shouldSpawnTTY) {
64
- const tty = await wm.spawnTTY({
65
- command: 'zsh',
66
- args: ['-lc', 'echo "window-manager lifecycle demo"; sleep 2; exit 7'],
67
- title: 'WM Lifecycle Demo',
68
- cols: 110,
69
- rows: 32,
70
- width: 960,
71
- height: 620,
72
- cwd: process.cwd(),
73
- })
74
- console.log('[spawn-tty]', tty)
75
- }
76
- })
77
-
78
- await container.sleep(150)
79
-
80
- console.log(`[ready] listening=${wm.isListening} socket=${wm.state.get('socketPath') ?? '<unset>'}`)
81
- if (!wm.isListening) {
82
- console.log(`[error] ${wm.state.get('lastError') ?? 'windowManager failed to bind socket'}`)
83
- console.log('[hint] try: bun run scripts/test-window-manager-lifecycle.ts --socket=/tmp/ipc-window.sock')
84
- }
85
- console.log('[usage] bun run scripts/test-window-manager-lifecycle.ts [--spawn-tty] [--spawn-window] [--socket=/path/to/ipc-window.sock]')
86
- console.log('[ready] waiting for native app connection...')
@@ -1,43 +0,0 @@
1
- import { NodeContainer } from '../src/node/container'
2
-
3
- const container = new NodeContainer({ cwd: process.cwd() })
4
- const wm = container.feature('windowManager')
5
-
6
- await wm.listen()
7
- console.log('Listening:', wm.isListening)
8
- console.log('Socket path:', wm.state.get('socketPath'))
9
-
10
- wm.on('clientConnected', () => {
11
- console.log('App connected')
12
- })
13
-
14
- wm.on('message', (msg) => {
15
- console.log('Message from app:', msg)
16
- })
17
-
18
- wm.on('clientDisconnected', () => {
19
- console.log('App disconnected')
20
- })
21
-
22
- // Wait for the app, then spawn a TTY running luca serve in the writing assistant playground
23
- wm.on('clientConnected', async () => {
24
- const result = await wm.spawnTTY({
25
- command: 'luca',
26
- args: ['serve', '--any-port'],
27
- cwd: '/Users/jon/@soederpop/playground/writing-assistant',
28
- title: 'Writing Assistant Server',
29
- cols: 120,
30
- rows: 40,
31
- width: 1000,
32
- height: 700,
33
- })
34
-
35
- console.log('Spawned TTY:', result)
36
-
37
- if (result.windowId) {
38
- console.log(`Window ID: ${result.windowId}`)
39
- console.log(`PID: ${result.pid}`)
40
- }
41
- })
42
-
43
- console.log('Waiting for native app to connect...')