@inetafrica/open-claudia 3.5.28 → 3.5.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -20
- package/core/actions.js +8 -0
- package/core/browser-control.js +17 -201
- package/core/browser-lock.js +2 -33
- package/core/browser-view.js +5 -90
- package/core/cards.js +0 -18
- package/core/chatter.js +41 -0
- package/core/enforcer.js +1 -1
- package/core/handlers.js +29 -3
- package/core/loopback.js +1 -4
- package/core/router.js +3 -2
- package/core/runner.js +7 -2
- package/core/tool-guard.js +0 -3
- package/core/turn-observer.js +18 -11
- package/package.json +2 -2
- package/test-enforcer.js +13 -0
- package/test-provider-language.js +9 -12
- package/test-thread-lanes.js +3 -0
- package/web.js +1 -4
- package/core/browser-drive.js +0 -619
package/CHANGELOG.md
CHANGED
|
@@ -1,30 +1,20 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 3.5.
|
|
3
|
+
## 3.5.47
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Your rules for a group now beat the guard's blanket "never share" line (Sumeet, 11 Oct 20:57 EAT: the network bot kept bringing NOC and Knock requests to him although his rules for both groups allow read-only diagnostics). Built on 3.5.33, the fleet's version; a canary for the network bot first.
|
|
6
6
|
|
|
7
|
-
-
|
|
8
|
-
- **
|
|
7
|
+
- **The guard judge's floor is scoped, not blanket** (`core/enforcer.js`). It said: whatever any layer says, never allow other people's private information or internal infrastructure details. A network bot's useful replies nearly always carry both (customer accounts, NAS names, IPs on blocklists), so the owner's channel rules never won and each reply became an owner card: 17 on the network bot on 11 Oct. Now credentials, secrets, passwords, tokens and keys stay blocked whatever any layer says; the owner's own private information needs a layer that explicitly allows it; infrastructure details and other people's information are allowed only where a channel rule or mandate clearly permits that kind of information or the work that produces it.
|
|
8
|
+
- **Proof:** two replies the network bot held on 11 Oct, replayed against the judge (claude-sonnet-5-5): with the old line 5 of 6 verdicts escalated, quoting it; with the new wording 4 of 4 allowed, and a made-up reply carrying a NAS password and RADIUS secret was blocked 2 of 2.
|
|
9
|
+
- **Tests:** `test-enforcer.js` checks the credentials floor is absolute and the blanket line is gone (it fails on the old wording).
|
|
9
10
|
|
|
10
|
-
## 3.5.
|
|
11
|
+
## 3.5.33
|
|
11
12
|
|
|
12
|
-
|
|
13
|
+
Quiet chats by default (Sumeet, 11 Oct 08:51 EAT, on Kazee unread counts: one voice note brought a "Heard:" echo, a progress message and "Recalled my notes" lines before the answer, each a new unread and often a push; 09:16 EAT "Yes" to turning them off by default on every bot). Built on 3.5.24, the fleet's version: the Browser canaries (3.5.25, 3.5.26, 3.5.28) and the call builds on Sumeet's own bot (3.5.27 and later) are not in it.
|
|
13
14
|
|
|
14
|
-
- **
|
|
15
|
-
- **
|
|
16
|
-
|
|
17
|
-
## 3.5.25
|
|
18
|
-
|
|
19
|
-
The Browser app feels like a browser on the phone and on the web (Sumeet, 10 Oct 08:53 EAT on his phone: "it feels like a browser on my phone… like Google Chrome app", and 09:18: "the web and mobile still need to also feel native… don't forget the website"). A canary build for the Browser canary bot first; the fleet waits for his OK.
|
|
20
|
-
|
|
21
|
-
- **The bot's tab takes the driver's screen.** With Chrome in its own container, a take (or a `fit` when the screen changes: a turned phone, a resized panel, a keyboard that opens) that carries the driver's screen (`viewport`: CSS size, pixel density, phone layout, touch) fits the bot's tab to it through `core/browser-drive.js`. A phone gets the page's phone layout, full screen and sharp; the web gets a desktop page at the panel's size. Handing back puts the tab back to its own size. The browser's identity (user agent) never changes, so sites signed into stay signed in after the hand back. `GET /api/browser/control` says `fit: true`.
|
|
22
|
-
- **The driver's input goes straight to Chrome,** through the view's own DevTools session, in order. Touches keep their timing, so a swipe flings; typed text is inserted whole (`input_text`, at most 500 characters: accents, emoji, the keyboard's corrections); every point is scaled by the view's density, because Chrome reads input in the drawn picture's pixels. A bot without the view takes `input_text` as key presses.
|
|
23
|
-
- **The page's text fields.** The view reads where the page's text fields are and what kind they are (never a value) and streams them as `fields` events, so the Browser page opens the phone's own keyboard of the right kind the moment a field is tapped.
|
|
24
|
-
- **Pictures.** The view streams its own screencast at the driver's size (at most 1.6 million pixels, 30 a second). `/api/browser/stream` serves it while someone drives, up to 30 fps, and keeps the newest frame while a slow viewer drains instead of dropping it, so the last picture of a scroll always arrives.
|
|
25
|
-
- **A hand back wakes the bot only when it waits for that person:** it sent them an Open Browser card (cards now record `openApp`), or one of its browser commands was refused while they drove (the tool guard marks it). Otherwise the hand back is only logged. On 9 Oct a test hand back set off a 28-minute tangent on the canary.
|
|
26
|
-
- **Correction to 3.5.24:** in container mode, sessions are not pinned to tabs (Agent Space dropped `AGENT_BROWSER_PIN_TAB` in a98f239); a session uses the tab that is open.
|
|
27
|
-
- **Tests:** `test-browser-drive.js` (new), `test-browser-control.js`, `test-web-browser-app.js`.
|
|
15
|
+
- **One message in, one answer out.** By default a turn now shows "typing" while it works and sends only its answer: no "Heard: …" echo of a voice note or audio file, no live progress message, and no "📖 Recalled my notes on …", "✏️ Updating my notes on …", new pack, entity, skill or "🧩 Reused" lines. Recall-failure alerts, /recall, /tooltrace, wake-up notices and limit warnings are unchanged.
|
|
16
|
+
- **Each comes back with its own command** (`core/chatter.js`): `/heard`, `/notes` and `/progress` (on, off, or no argument for the current value and buttons), saved per person like /verbose. `/verbose` streams thinking into the progress message, so it turns the progress message on too: anyone with /verbose on keeps it until they turn /verbose off.
|
|
17
|
+
- **Tests:** `test-quiet-defaults.js` (new, in `npm test`), `test-thread-lanes.js` (its live-preview check turns /progress on).
|
|
28
18
|
|
|
29
19
|
## 3.5.24
|
|
30
20
|
|
package/core/actions.js
CHANGED
|
@@ -6,6 +6,7 @@ const fs = require("fs");
|
|
|
6
6
|
const path = require("path");
|
|
7
7
|
const { CONFIG_DIR, WORKSPACE, CHAT_ID } = require("./config");
|
|
8
8
|
const { send } = require("./io");
|
|
9
|
+
const { TOGGLES: chatterToggles } = require("./chatter");
|
|
9
10
|
const {
|
|
10
11
|
currentState, saveState,
|
|
11
12
|
getActiveProvider, getProjectKey, consumeSessionCallbackToken,
|
|
@@ -959,6 +960,13 @@ async function handleAction(envelope) {
|
|
|
959
960
|
await send(`Verbose thinking: ${state.settings.showThinking ? "on" : "off"}`);
|
|
960
961
|
return;
|
|
961
962
|
}
|
|
963
|
+
const chatterToggle = chatterToggles.find((toggle) => d.startsWith(`${toggle.callback}:`));
|
|
964
|
+
if (chatterToggle) {
|
|
965
|
+
state.settings[chatterToggle.key] = d.slice(chatterToggle.callback.length + 1) === "on";
|
|
966
|
+
saveState();
|
|
967
|
+
await send(`${chatterToggle.label}: ${state.settings[chatterToggle.key] ? "on" : "off"}`);
|
|
968
|
+
return;
|
|
969
|
+
}
|
|
962
970
|
if (d.startsWith("cw:")) {
|
|
963
971
|
const v = d.slice(3);
|
|
964
972
|
if (v === "default") state.settings.compactWindow = null;
|
package/core/browser-control.js
CHANGED
|
@@ -13,30 +13,18 @@
|
|
|
13
13
|
// after the input before them, with the bot's own agent-browser environment and never a
|
|
14
14
|
// launch flag (core/agent-browser-cli.js: other launch settings would restart the browser).
|
|
15
15
|
// Only public internet addresses (core/browser-address.js).
|
|
16
|
-
// - the phone and web view (core/browser-drive.js): a take or a "fit" with the driver's screen
|
|
17
|
-
// makes the bot's tab take its size, phone layout and touch, and then the driver's input goes
|
|
18
|
-
// straight to Chrome through that view (typed text too: input_text). Only with Chrome in its
|
|
19
|
-
// own container; elsewhere the browser keeps its size and input goes through agent-browser.
|
|
20
|
-
// - always on: a person who opens the Browser app while the bot's browser is not open gets it
|
|
21
|
-
// opened on a start page, in the bot's own default session (the one its `agent-browser open`
|
|
22
|
-
// uses), with the bot's own agent-browser environment, so the bot carries on in that tab.
|
|
23
16
|
// - hand back (release, a closed panel, or a lock found expired by the 30 s sweeper): the lock
|
|
24
|
-
// goes, the
|
|
25
|
-
//
|
|
26
|
-
// or one of its browser commands was refused while they drove. A recording stopped at the
|
|
27
|
-
// same time folds into one wake-up. Otherwise the hand back is only logged.
|
|
17
|
+
// goes, the socket closes, any recording stops, and the bot is woken in the person's own chat
|
|
18
|
+
// to carry on (C6). A recording stopped at the same time folds into one wake-up.
|
|
28
19
|
//
|
|
29
20
|
// Every route answers { status, body }; web.js writes it. Agent Space reaches these routes with
|
|
30
21
|
// the control token only.
|
|
31
22
|
|
|
32
|
-
const os = require("os");
|
|
33
|
-
const path = require("path");
|
|
34
23
|
const WebSocket = require("ws");
|
|
35
24
|
const lockFile = require("./browser-lock");
|
|
36
25
|
const view = require("./browser-view");
|
|
37
26
|
const teach = require("./browser-teach");
|
|
38
27
|
const address = require("./browser-address");
|
|
39
|
-
const drive = require("./browser-drive");
|
|
40
28
|
const { runAgentBrowser } = require("./agent-browser-cli");
|
|
41
29
|
|
|
42
30
|
const LOCK_TTL_MS = lockFile.LOCK_TTL_MS;
|
|
@@ -53,13 +41,6 @@ const ID_RE = /^[A-Za-z0-9._:-]{1,100}$/;
|
|
|
53
41
|
// An address-bar command still running after this is killed, and the person told it failed.
|
|
54
42
|
const NAVIGATE_TIMEOUT_MS = 20 * 1000;
|
|
55
43
|
const NAVIGATIONS = new Set(["open", "back", "forward", "reload"]);
|
|
56
|
-
// Always on: the page a browser opened for a person starts on, and the session it opens (the one
|
|
57
|
-
// the bot's own `agent-browser` commands use when they name none).
|
|
58
|
-
const START_PAGE = "https://www.google.com/";
|
|
59
|
-
const START_SESSION = "default";
|
|
60
|
-
|
|
61
|
-
// An Open Browser card counts as a request for a person this long.
|
|
62
|
-
const REQUEST_MAX_AGE_MS = 24 * 60 * 60 * 1000;
|
|
63
44
|
|
|
64
45
|
const HAND_BACK_REASONS = {
|
|
65
46
|
closed: " (they closed the Browser)",
|
|
@@ -126,7 +107,6 @@ const MOUSE_EVENTS = new Set(["mousePressed", "mouseReleased", "mouseMoved", "mo
|
|
|
126
107
|
const MOUSE_BUTTONS = new Set(["none", "left", "middle", "right"]);
|
|
127
108
|
const KEY_EVENTS = new Set(["keyDown", "keyUp", "char"]);
|
|
128
109
|
const TOUCH_EVENTS = new Set(["touchStart", "touchMove", "touchEnd", "touchCancel"]);
|
|
129
|
-
const TEXT_MAX = drive.TEXT_MAX;
|
|
130
110
|
|
|
131
111
|
const isNumber = (v) => typeof v === "number" && Number.isFinite(v);
|
|
132
112
|
const inRange = (v, lo, hi) => isNumber(v) && v >= lo && v <= hi;
|
|
@@ -183,34 +163,11 @@ function checkEvent(ev) {
|
|
|
183
163
|
}
|
|
184
164
|
const out = { type: "input_touch", eventType: ev.eventType, touchPoints };
|
|
185
165
|
if (!optional(out, "modifiers", modifiers)) return null;
|
|
186
|
-
// When the finger moved, on the driver's device (ms): keeps a swipe's speed (core/browser-drive.js).
|
|
187
|
-
if (!optional(out, "t", (v) => inRange(v, 0, 1e13))) return null;
|
|
188
166
|
return out;
|
|
189
167
|
}
|
|
190
|
-
// Text the driver typed (their phone keyboard, a paste), inserted where the page's caret is.
|
|
191
|
-
if (ev.type === "input_text") {
|
|
192
|
-
if (typeof ev.text !== "string" || !ev.text || Array.from(ev.text).length > TEXT_MAX) return null;
|
|
193
|
-
return { type: "input_text", text: ev.text };
|
|
194
|
-
}
|
|
195
168
|
return null;
|
|
196
169
|
}
|
|
197
170
|
|
|
198
|
-
// input_text as key presses, for a bot without the driver's view (agent-browser's socket takes
|
|
199
|
-
// no text) and for Teach me, which counts typing from key presses.
|
|
200
|
-
function textAsKeys(text) {
|
|
201
|
-
const out = [];
|
|
202
|
-
for (const ch of Array.from(text)) {
|
|
203
|
-
if (ch === "\n" || ch === "\r") {
|
|
204
|
-
out.push({ type: "input_keyboard", eventType: "keyDown", key: "Enter", code: "Enter", text: "\r", windowsVirtualKeyCode: 13 });
|
|
205
|
-
out.push({ type: "input_keyboard", eventType: "keyUp", key: "Enter", code: "Enter", windowsVirtualKeyCode: 13 });
|
|
206
|
-
} else {
|
|
207
|
-
out.push({ type: "input_keyboard", eventType: "keyDown", key: ch, text: ch });
|
|
208
|
-
out.push({ type: "input_keyboard", eventType: "keyUp", key: ch });
|
|
209
|
-
}
|
|
210
|
-
}
|
|
211
|
-
return out;
|
|
212
|
-
}
|
|
213
|
-
|
|
214
171
|
// ---------------------------------------------------------------------------
|
|
215
172
|
// Input sockets: one per agent-browser session, opened on take (or on the first input after a
|
|
216
173
|
// restart), closed on hand back.
|
|
@@ -368,23 +325,7 @@ function wakeChat({ chatId, userId }, prompt, label) {
|
|
|
368
325
|
|
|
369
326
|
function handBackLine(name, reason) {
|
|
370
327
|
const who = teach.cleanName(name) || "Someone";
|
|
371
|
-
return `[Browser] ${who} handed the browser back${HAND_BACK_REASONS[reason] || ""}.
|
|
372
|
-
}
|
|
373
|
-
|
|
374
|
-
// Was the bot waiting for this person when they handed back? It asked for them with an Open
|
|
375
|
-
// Browser card in their chat (since the day before they took over), or one of its own browser
|
|
376
|
-
// commands was refused while they drove.
|
|
377
|
-
function botWasWaiting(lock) {
|
|
378
|
-
let asked = [];
|
|
379
|
-
try {
|
|
380
|
-
const since = Date.parse(lock.since);
|
|
381
|
-
asked = require("./cards").takeBrowserRequests(lock.chatId, (Number.isFinite(since) ? since : deps.now()) - REQUEST_MAX_AGE_MS, deps.now());
|
|
382
|
-
} catch (e) {
|
|
383
|
-
console.error(`[browser] reading the bot's Open Browser cards failed: ${e.message}`);
|
|
384
|
-
}
|
|
385
|
-
const refused = lockFile.waitedDuring(lock);
|
|
386
|
-
lockFile.clearWaiting();
|
|
387
|
-
return asked.length > 0 || refused;
|
|
328
|
+
return `[Browser] ${who} handed the browser back${HAND_BACK_REASONS[reason] || ""}. If you were waiting on them, check the page first (it may have changed) and carry on; otherwise just note it.`;
|
|
388
329
|
}
|
|
389
330
|
|
|
390
331
|
// ---------------------------------------------------------------------------
|
|
@@ -394,39 +335,7 @@ function lockBody(lock) {
|
|
|
394
335
|
return { driving: true, driver: { name: lock.name } };
|
|
395
336
|
}
|
|
396
337
|
|
|
397
|
-
|
|
398
|
-
// in the lock, so a restarted bot can bring it back. -> the fit, or null (the browser keeps its size).
|
|
399
|
-
async function startView(lock, viewport) {
|
|
400
|
-
if (!viewport || !drive.supported()) return null;
|
|
401
|
-
const info = sessionInfo(lock.session);
|
|
402
|
-
if (!info) return null;
|
|
403
|
-
const r = drive.active(lock.session)
|
|
404
|
-
? await drive.fit(viewport)
|
|
405
|
-
: await drive.start({ session: info, viewport, savedBounds: lock.bounds || null });
|
|
406
|
-
const now = lockFile.readLock();
|
|
407
|
-
// The lock changed hands (or was handed back) while the view started: undo it.
|
|
408
|
-
if (!now || now.since !== lock.since || now.userId !== lock.userId) {
|
|
409
|
-
if (r.ok && drive.active(lock.session)) await drive.stop();
|
|
410
|
-
return null;
|
|
411
|
-
}
|
|
412
|
-
if (!r.ok) return null;
|
|
413
|
-
now.fit = r.fit;
|
|
414
|
-
if (r.bounds) now.bounds = r.bounds;
|
|
415
|
-
lockFile.writeLock(now);
|
|
416
|
-
return r.fit;
|
|
417
|
-
}
|
|
418
|
-
|
|
419
|
-
// After a bot restart the lock remembers a view that is no longer live: bring it back. Once at a
|
|
420
|
-
// time, in the background.
|
|
421
|
-
let reviving = null;
|
|
422
|
-
function reviveView(lock) {
|
|
423
|
-
if (!lock || !lock.fit || reviving || drive.active(lock.session) || !drive.supported()) return;
|
|
424
|
-
reviving = startView(lock, lock.fit)
|
|
425
|
-
.catch((e) => console.error(`[browser] bringing the driver's view back failed: ${e.message}`))
|
|
426
|
-
.finally(() => { reviving = null; });
|
|
427
|
-
}
|
|
428
|
-
|
|
429
|
-
async function take({ session, who, viewport } = {}) {
|
|
338
|
+
async function take({ session, who } = {}) {
|
|
430
339
|
if (!validSession(session) || !validWho(who, { needChat: true })) return reply(400, { error: "bad-request" });
|
|
431
340
|
// A lock that ran out but has not been swept yet is handed back first, the sweeper's way: its
|
|
432
341
|
// driver's chat is woken and their recording stopped, whoever takes over now.
|
|
@@ -445,25 +354,9 @@ async function take({ session, who, viewport } = {}) {
|
|
|
445
354
|
since: current && current.session === session ? current.since : iso(now),
|
|
446
355
|
expiresAt: iso(now + LOCK_TTL_MS),
|
|
447
356
|
};
|
|
448
|
-
// The same person taking again (a reloaded page) keeps the window's size from before their view.
|
|
449
|
-
if (current && current.session === session && sameUser(current, who) && current.bounds) lock.bounds = current.bounds;
|
|
450
357
|
lockFile.writeLock(lock);
|
|
451
358
|
openInputSocket(session);
|
|
452
|
-
|
|
453
|
-
// fit: the driver's view now live, or null (this bot's browser keeps its own size).
|
|
454
|
-
const fitted = await startView(lock, viewport);
|
|
455
|
-
return reply(200, { ...lockBody(lock), fit: fitted });
|
|
456
|
-
}
|
|
457
|
-
|
|
458
|
-
// The driver's screen changed (a turned phone, a resized panel): { session, who, viewport }.
|
|
459
|
-
async function fitView({ session, who, viewport } = {}) {
|
|
460
|
-
if (!validSession(session) || !validWho(who)) return reply(400, { error: "bad-request" });
|
|
461
|
-
if (!drive.checkViewport(viewport)) return reply(400, { error: "bad-viewport" });
|
|
462
|
-
const lock = extendDriverLock(session, who);
|
|
463
|
-
if (!lock) return reply(409, { driving: false, error: "not-driving" });
|
|
464
|
-
if (!drive.supported()) return reply(200, { ...lockBody(lock), fit: null });
|
|
465
|
-
const fitted = await startView(lock, viewport);
|
|
466
|
-
return reply(200, { ...lockBody(lock), fit: fitted });
|
|
359
|
+
return reply(200, lockBody(lock));
|
|
467
360
|
}
|
|
468
361
|
|
|
469
362
|
function renew({ session, who } = {}) {
|
|
@@ -476,8 +369,7 @@ function renew({ session, who } = {}) {
|
|
|
476
369
|
current.expiresAt = iso(now + LOCK_TTL_MS);
|
|
477
370
|
lockFile.writeLock(current);
|
|
478
371
|
openInputSocket(current.session); // after a restart the socket comes back here
|
|
479
|
-
|
|
480
|
-
return reply(200, current.fit ? { ...lockBody(current), fit: drive.active(current.session) ? current.fit : null } : lockBody(current));
|
|
372
|
+
return reply(200, lockBody(current));
|
|
481
373
|
}
|
|
482
374
|
|
|
483
375
|
// Hand the browser back: claim the lock (synchronously, so a release and the sweeper never both
|
|
@@ -491,13 +383,6 @@ async function handBack(expected, { reason = null } = {}) {
|
|
|
491
383
|
let stopped = null;
|
|
492
384
|
await runQueued(current.session, 0, async () => {
|
|
493
385
|
closeInputSocket(current.session);
|
|
494
|
-
// The bot's tab goes back to its own size before the bot carries on.
|
|
495
|
-
try {
|
|
496
|
-
if (drive.active(current.session)) await drive.stop();
|
|
497
|
-
else if (current.bounds) await drive.restoreWindow(info, current.bounds);
|
|
498
|
-
} catch (e) {
|
|
499
|
-
console.error(`[browser] ending the driver's view failed: ${e.message}`);
|
|
500
|
-
}
|
|
501
386
|
try {
|
|
502
387
|
stopped = await teach.stopRecording(current.userId, { session: info });
|
|
503
388
|
} catch (e) {
|
|
@@ -505,10 +390,9 @@ async function handBack(expected, { reason = null } = {}) {
|
|
|
505
390
|
}
|
|
506
391
|
});
|
|
507
392
|
const line = handBackLine(current.name, reason);
|
|
508
|
-
const waited = botWasWaiting(current);
|
|
509
393
|
if (stopped && !stopped.discarded) wakeChat(current, teach.teachPrompt(stopped, { handBack: line }), "Teach me");
|
|
510
|
-
else
|
|
511
|
-
console.log(`[browser] ${current.userId} handed the browser back${reason ? ` (${reason})` : ""}
|
|
394
|
+
else wakeChat(current, line, "Browser");
|
|
395
|
+
console.log(`[browser] ${current.userId} handed the browser back${reason ? ` (${reason})` : ""}`);
|
|
512
396
|
return true;
|
|
513
397
|
}
|
|
514
398
|
|
|
@@ -525,13 +409,10 @@ async function control(body = {}) {
|
|
|
525
409
|
if (body.action === "take") return take(body);
|
|
526
410
|
if (body.action === "renew") return renew(body);
|
|
527
411
|
if (body.action === "release") return release(body);
|
|
528
|
-
if (body.action === "fit") return fitView(body);
|
|
529
412
|
return reply(400, { error: "bad-action" });
|
|
530
413
|
}
|
|
531
414
|
|
|
532
|
-
// navigate: true tells the front door this bot has the address bar (POST /api/browser/navigate)
|
|
533
|
-
// fit: true that a take can bring the driver's view (core/browser-drive.js) and input_text works;
|
|
534
|
-
// start: true that it opens its browser on a start page when there is none (POST /api/browser/start).
|
|
415
|
+
// navigate: true tells the front door this bot has the address bar (POST /api/browser/navigate).
|
|
535
416
|
function status() {
|
|
536
417
|
const lock = lockFile.liveLock(deps.now());
|
|
537
418
|
return reply(200, {
|
|
@@ -539,50 +420,9 @@ function status() {
|
|
|
539
420
|
driver: lock ? { name: lock.name, userId: lock.userId } : null,
|
|
540
421
|
recording: teach.activeSummary(),
|
|
541
422
|
navigate: true,
|
|
542
|
-
fit: drive.supported(),
|
|
543
|
-
start: true,
|
|
544
423
|
});
|
|
545
424
|
}
|
|
546
425
|
|
|
547
|
-
// ---------------------------------------------------------------------------
|
|
548
|
-
// Always on: POST /api/browser/start
|
|
549
|
-
|
|
550
|
-
// The socket folder the bot's own agent-browser commands use (its agent runs without
|
|
551
|
-
// XDG_RUNTIME_DIR): AGENT_BROWSER_SOCKET_DIR, else ~/.agent-browser.
|
|
552
|
-
function agentSocketDir(env = process.env, home = os.homedir()) {
|
|
553
|
-
if (env.AGENT_BROWSER_SOCKET_DIR) return env.AGENT_BROWSER_SOCKET_DIR;
|
|
554
|
-
return home ? path.join(home, ".agent-browser") : null;
|
|
555
|
-
}
|
|
556
|
-
|
|
557
|
-
let starting = null;
|
|
558
|
-
|
|
559
|
-
// { who: { userId } }. A browser already open is answered as it is (the newest session); otherwise
|
|
560
|
-
// the bot's default session opens on the start page, as the bot's own `agent-browser open` would,
|
|
561
|
-
// and two people opening the app at once start one browser. { session, started }.
|
|
562
|
-
async function start({ who } = {}) {
|
|
563
|
-
if (!validWho(who)) return reply(400, { error: "bad-request" });
|
|
564
|
-
const open = deps.listSessions();
|
|
565
|
-
if (open.length) return reply(200, { session: open[0].name, started: false });
|
|
566
|
-
if (!starting) {
|
|
567
|
-
starting = (async () => {
|
|
568
|
-
const dir = agentSocketDir();
|
|
569
|
-
const run = await runAgentBrowser({ name: START_SESSION, dir }, ["open", START_PAGE], {
|
|
570
|
-
timeoutMs: deps.navigateTimeoutMs,
|
|
571
|
-
execFile: deps.execFile || undefined,
|
|
572
|
-
});
|
|
573
|
-
const listed = run.ok ? deps.listSessions().find((s) => s.name === START_SESSION) : null;
|
|
574
|
-
if (!listed) {
|
|
575
|
-
const why = run.timedOut ? `agent-browser took longer than ${Math.ceil(deps.navigateTimeoutMs / 1000)} s and was stopped` : run.ok ? "the browser did not come up" : "agent-browser failed";
|
|
576
|
-
console.error(`[browser] could not open the browser for ${who.userId}: ${why}`);
|
|
577
|
-
return reply(503, { error: "browser-unavailable" });
|
|
578
|
-
}
|
|
579
|
-
console.log(`[browser] opened the browser on the start page for ${who.userId}`);
|
|
580
|
-
return reply(200, { session: listed.name, started: true });
|
|
581
|
-
})().finally(() => { starting = null; });
|
|
582
|
-
}
|
|
583
|
-
return starting;
|
|
584
|
-
}
|
|
585
|
-
|
|
586
426
|
// ---------------------------------------------------------------------------
|
|
587
427
|
// C3: input
|
|
588
428
|
|
|
@@ -596,41 +436,24 @@ async function input({ session, events, who } = {}) {
|
|
|
596
436
|
messages.push(msg);
|
|
597
437
|
}
|
|
598
438
|
// Every accepted input extends the lock.
|
|
599
|
-
|
|
600
|
-
if (!lock) return reply(409, { error: "not-driving" });
|
|
439
|
+
if (!extendDriverLock(session, who)) return reply(409, { error: "not-driving" });
|
|
601
440
|
if (!messages.length) return reply(200, { ok: true, accepted: 0 });
|
|
602
|
-
if (lock.fit && !drive.active(session)) reviveView(lock);
|
|
603
441
|
const entry = openInputSocket(session);
|
|
604
|
-
if (!entry
|
|
442
|
+
if (!entry) return reply(404, { error: "not-running" });
|
|
605
443
|
const job = runQueued(session, messages.length, async () => {
|
|
444
|
+
if (!(await socketReady(entry))) return reply(503, { error: "browser-unavailable", accepted: 0 });
|
|
606
445
|
const recorder = teach.recorderFor(session, who.userId);
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
for (const ev of msg.type === "input_text" ? textAsKeys(msg.text) : [msg]) {
|
|
446
|
+
let sent = 0;
|
|
447
|
+
for (const msg of messages) {
|
|
448
|
+
if (recorder) {
|
|
611
449
|
try {
|
|
612
|
-
await recorder.onEvent(
|
|
450
|
+
await recorder.onEvent(msg, { session: { name: entry.name, dir: entry.dir }, frame: entry.frame });
|
|
613
451
|
} catch (e) {
|
|
614
452
|
console.error(`[browser] recording a step failed: ${e.message}`);
|
|
615
453
|
}
|
|
616
454
|
}
|
|
617
|
-
};
|
|
618
|
-
// The driver's view: straight to Chrome, in order, without waiting for each answer.
|
|
619
|
-
if (drive.active(session)) {
|
|
620
|
-
const info = sessionInfo(session) || { name: session, dir: entry ? entry.dir : null };
|
|
621
|
-
for (const msg of messages) {
|
|
622
|
-
await record(msg, drive.lastFrame(), { name: info.name, dir: info.dir });
|
|
623
|
-
if (drive.dispatch([msg]) < 0) return reply(503, { error: "browser-unavailable" });
|
|
624
|
-
}
|
|
625
|
-
return reply(200, { ok: true, accepted: messages.length });
|
|
626
|
-
}
|
|
627
|
-
if (!entry || !(await socketReady(entry))) return reply(503, { error: "browser-unavailable", accepted: 0 });
|
|
628
|
-
let sent = 0;
|
|
629
|
-
for (const msg of messages) {
|
|
630
|
-
await record(msg, entry.frame, { name: entry.name, dir: entry.dir });
|
|
631
455
|
if (!entry.open || entry.closed) return reply(503, { error: "browser-unavailable", accepted: sent });
|
|
632
|
-
|
|
633
|
-
for (const ev of msg.type === "input_text" ? textAsKeys(msg.text) : [msg]) entry.ws.send(JSON.stringify(ev));
|
|
456
|
+
entry.ws.send(JSON.stringify(msg));
|
|
634
457
|
sent += 1;
|
|
635
458
|
}
|
|
636
459
|
return reply(200, { ok: true, accepted: sent });
|
|
@@ -779,8 +602,6 @@ function _reset() {
|
|
|
779
602
|
for (const name of [...sockets.keys()]) closeInputSocket(name);
|
|
780
603
|
queues.clear();
|
|
781
604
|
if (sweepTimer) { clearInterval(sweepTimer); sweepTimer = null; }
|
|
782
|
-
reviving = null;
|
|
783
|
-
drive._reset();
|
|
784
605
|
deps = defaultDeps();
|
|
785
606
|
}
|
|
786
607
|
|
|
@@ -788,20 +609,15 @@ module.exports = {
|
|
|
788
609
|
LOCK_TTL_MS,
|
|
789
610
|
MAX_EVENTS,
|
|
790
611
|
NAVIGATE_TIMEOUT_MS,
|
|
791
|
-
START_PAGE,
|
|
792
|
-
START_SESSION,
|
|
793
612
|
SWEEP_INTERVAL_MS,
|
|
794
|
-
agentSocketDir,
|
|
795
613
|
checkEvent,
|
|
796
614
|
chatOrigin,
|
|
797
615
|
control,
|
|
798
|
-
fitView,
|
|
799
616
|
handBackLine,
|
|
800
617
|
input,
|
|
801
618
|
navigate,
|
|
802
619
|
release,
|
|
803
620
|
renew,
|
|
804
|
-
start,
|
|
805
621
|
startSweeper,
|
|
806
622
|
status,
|
|
807
623
|
sweepLock,
|
package/core/browser-lock.js
CHANGED
|
@@ -4,9 +4,7 @@
|
|
|
4
4
|
// hook (core/tool-guard.js, C7), which runs as a separate process for every tool call and
|
|
5
5
|
// re-reads it each time, like the tooling-mode switch (core/tooling-mode.js).
|
|
6
6
|
//
|
|
7
|
-
// { session, userId, name, chatId, since, expiresAt
|
|
8
|
-
// fit: the driver's view (core/browser-drive.js), bounds: the window's size before it, so a
|
|
9
|
-
// restarted bot can bring the view back, and the hand back can put the window back.
|
|
7
|
+
// { session, userId, name, chatId, since, expiresAt } (ISO times; written atomically, 0600)
|
|
10
8
|
//
|
|
11
9
|
// An expired lock is no lock: it never blocks the bot, and the sweeper in
|
|
12
10
|
// core/browser-control.js hands the browser back.
|
|
@@ -50,33 +48,4 @@ function removeLock() {
|
|
|
50
48
|
try { fs.rmSync(driverFile(), { force: true }); } catch (_) { /* already gone */ }
|
|
51
49
|
}
|
|
52
50
|
|
|
53
|
-
|
|
54
|
-
// this lock was live), so the hand back must wake it. Keyed by the lock's `since`: a marker from
|
|
55
|
-
// an earlier drive never counts. A separate file, because the tool guard runs in its own process
|
|
56
|
-
// and the lock file is rewritten every second while someone drives.
|
|
57
|
-
function waitingFile() {
|
|
58
|
-
return process.env.OC_BROWSER_WAITING_FILE || path.join(require("../config-dir"), "browser-waiting.json");
|
|
59
|
-
}
|
|
60
|
-
|
|
61
|
-
function markWaiting(lock) {
|
|
62
|
-
if (!lock || !lock.since) return;
|
|
63
|
-
try {
|
|
64
|
-
atomicWriteFileSync(waitingFile(), `${JSON.stringify({ since: lock.since, at: new Date().toISOString() })}\n`, { mode: 0o600 });
|
|
65
|
-
} catch (_) { /* best effort: the bot then just is not woken */ }
|
|
66
|
-
}
|
|
67
|
-
|
|
68
|
-
function waitedDuring(lock) {
|
|
69
|
-
if (!lock || !lock.since) return false;
|
|
70
|
-
try {
|
|
71
|
-
const mark = JSON.parse(fs.readFileSync(waitingFile(), "utf8"));
|
|
72
|
-
return !!mark && mark.since === lock.since;
|
|
73
|
-
} catch (_) {
|
|
74
|
-
return false;
|
|
75
|
-
}
|
|
76
|
-
}
|
|
77
|
-
|
|
78
|
-
function clearWaiting() {
|
|
79
|
-
try { fs.rmSync(waitingFile(), { force: true }); } catch (_) { /* already gone */ }
|
|
80
|
-
}
|
|
81
|
-
|
|
82
|
-
module.exports = { LOCK_TTL_MS, driverFile, readLock, lockExpiresAt, liveLock, writeLock, removeLock, markWaiting, waitedDuring, clearWaiting };
|
|
51
|
+
module.exports = { LOCK_TTL_MS, driverFile, readLock, lockExpiresAt, liveLock, writeLock, removeLock };
|
package/core/browser-view.js
CHANGED
|
@@ -22,22 +22,11 @@
|
|
|
22
22
|
const fs = require("fs");
|
|
23
23
|
const os = require("os");
|
|
24
24
|
const path = require("path");
|
|
25
|
-
const { EventEmitter } = require("events");
|
|
26
25
|
const WebSocket = require("ws");
|
|
27
26
|
|
|
28
27
|
const SESSION_NAME = /^[A-Za-z0-9._-]{1,64}$/;
|
|
29
28
|
const DEFAULT_FPS = 5;
|
|
30
|
-
const MAX_FPS =
|
|
31
|
-
|
|
32
|
-
// The phone and web view (core/browser-drive.js) publishes here while a person drives: "frame",
|
|
33
|
-
// "fields" (the page's text fields) and "start" / "stop", each with the session's name. A viewer
|
|
34
|
-
// of that session gets these instead of agent-browser's own frames, which follow neither the
|
|
35
|
-
// driver's size nor the page's real size. `active` and `last` let a viewer that opens mid-drive
|
|
36
|
-
// start with the picture at once.
|
|
37
|
-
const driveFeed = new EventEmitter();
|
|
38
|
-
driveFeed.setMaxListeners(200);
|
|
39
|
-
driveFeed.active = null;
|
|
40
|
-
driveFeed.last = null;
|
|
29
|
+
const MAX_FPS = 10;
|
|
41
30
|
const KEEPALIVE_MS = 25000;
|
|
42
31
|
const ACTIVATE_TIMEOUT_MS = 1500;
|
|
43
32
|
const TARGET_ID = /^[A-Za-z0-9]{8,64}$/; // static pages send no frames; comments keep proxies from closing the stream
|
|
@@ -224,77 +213,18 @@ function openStream(sessionName, { fps, sessions, connect = (url) => new WebSock
|
|
|
224
213
|
|
|
225
214
|
// Server-sent events for one session: "frame" and "url" events, a final "end" event with the
|
|
226
215
|
// reason, and ": ping" comments while the page is still. Every payload also carries its type,
|
|
227
|
-
// for readers that only look at data lines.
|
|
228
|
-
//
|
|
229
|
-
|
|
230
|
-
// frames are the driver's (core/browser-drive.js), at most `fps` a second, with "fields" events
|
|
231
|
-
// (the page's text fields) and "fit" events (the driver's view started, changed or ended).
|
|
232
|
-
function serveStream(res, sessionName, { fps, sessions, connect, activate, keepaliveMs = KEEPALIVE_MS, feed = driveFeed } = {}) {
|
|
216
|
+
// for readers that only look at data lines. Frames are skipped, never queued, while a slow
|
|
217
|
+
// viewer's connection is backed up.
|
|
218
|
+
function serveStream(res, sessionName, { fps, sessions, connect, activate, keepaliveMs = KEEPALIVE_MS } = {}) {
|
|
233
219
|
res.writeHead(200, {
|
|
234
220
|
"Content-Type": "text/event-stream",
|
|
235
221
|
"Cache-Control": "no-cache, no-store",
|
|
236
222
|
"Connection": "keep-alive",
|
|
237
223
|
"X-Accel-Buffering": "no",
|
|
238
224
|
});
|
|
239
|
-
let closed = false;
|
|
240
225
|
const write = (event, obj) => {
|
|
241
|
-
if (closed) return;
|
|
242
226
|
try { res.write(`event: ${event}\ndata: ${JSON.stringify(obj)}\n\n`); } catch (_) { /* viewer gone */ }
|
|
243
227
|
};
|
|
244
|
-
const interval = 1000 / clampFps(fps);
|
|
245
|
-
let lastSent = 0;
|
|
246
|
-
let pending = null;
|
|
247
|
-
let slot = null;
|
|
248
|
-
const flush = () => {
|
|
249
|
-
if (!pending || closed || res.writableNeedDrain) return;
|
|
250
|
-
const now = Date.now();
|
|
251
|
-
const due = lastSent + interval;
|
|
252
|
-
if (now < due) {
|
|
253
|
-
if (!slot) {
|
|
254
|
-
slot = setTimeout(() => { slot = null; flush(); }, due - now);
|
|
255
|
-
if (slot.unref) slot.unref();
|
|
256
|
-
}
|
|
257
|
-
return;
|
|
258
|
-
}
|
|
259
|
-
const msg = pending;
|
|
260
|
-
pending = null;
|
|
261
|
-
lastSent = now;
|
|
262
|
-
write("frame", msg);
|
|
263
|
-
};
|
|
264
|
-
const offerFrame = (msg) => {
|
|
265
|
-
pending = msg;
|
|
266
|
-
flush();
|
|
267
|
-
};
|
|
268
|
-
if (typeof res.on === "function") res.on("drain", flush);
|
|
269
|
-
let driving = feed.active === sessionName;
|
|
270
|
-
const onDriveFrame = (ev) => {
|
|
271
|
-
if (ev.session !== sessionName || !driving) return;
|
|
272
|
-
offerFrame({ type: "frame", seq: ev.frame.seq, data: ev.frame.data, metadata: ev.frame.metadata });
|
|
273
|
-
};
|
|
274
|
-
const onDriveFields = (ev) => {
|
|
275
|
-
if (ev.session !== sessionName || !driving) return;
|
|
276
|
-
write("fields", { type: "fields", fields: ev.fields, focused: ev.focused, editable: ev.editable });
|
|
277
|
-
};
|
|
278
|
-
const onDriveStart = (ev) => {
|
|
279
|
-
if (ev.session !== sessionName) return;
|
|
280
|
-
driving = true;
|
|
281
|
-
write("fit", { type: "fit", fit: ev.fit || null });
|
|
282
|
-
};
|
|
283
|
-
const onDriveStop = (ev) => {
|
|
284
|
-
if (ev.session !== sessionName) return;
|
|
285
|
-
driving = false;
|
|
286
|
-
write("fit", { type: "fit", fit: null });
|
|
287
|
-
};
|
|
288
|
-
feed.on("frame", onDriveFrame);
|
|
289
|
-
feed.on("fields", onDriveFields);
|
|
290
|
-
feed.on("start", onDriveStart);
|
|
291
|
-
feed.on("stop", onDriveStop);
|
|
292
|
-
const unsubscribe = () => {
|
|
293
|
-
feed.removeListener("frame", onDriveFrame);
|
|
294
|
-
feed.removeListener("fields", onDriveFields);
|
|
295
|
-
feed.removeListener("start", onDriveStart);
|
|
296
|
-
feed.removeListener("stop", onDriveStop);
|
|
297
|
-
};
|
|
298
228
|
const heartbeat = setInterval(() => { try { res.write(": ping\n\n"); } catch (_) { /* viewer gone */ } }, keepaliveMs);
|
|
299
229
|
const stream = openStream(sessionName, {
|
|
300
230
|
fps,
|
|
@@ -302,30 +232,17 @@ function serveStream(res, sessionName, { fps, sessions, connect, activate, keepa
|
|
|
302
232
|
connect,
|
|
303
233
|
...(activate ? { activate } : {}),
|
|
304
234
|
onMessage: (msg) => {
|
|
305
|
-
if (msg.type === "frame")
|
|
306
|
-
if (!driving) offerFrame(msg);
|
|
307
|
-
return;
|
|
308
|
-
}
|
|
235
|
+
if (msg.type === "frame" && res.writableNeedDrain) return;
|
|
309
236
|
write(msg.type, msg);
|
|
310
237
|
},
|
|
311
238
|
onEnd: (reason) => {
|
|
312
239
|
clearInterval(heartbeat);
|
|
313
|
-
unsubscribe();
|
|
314
240
|
write("end", { type: "end", reason });
|
|
315
|
-
closed = true;
|
|
316
241
|
try { res.end(); } catch (_) { /* viewer gone */ }
|
|
317
242
|
},
|
|
318
243
|
});
|
|
319
|
-
// A viewer that opens while someone drives starts with the driver's picture.
|
|
320
|
-
if (driving && feed.last && feed.last.session === sessionName) {
|
|
321
|
-
const f = feed.last.frame;
|
|
322
|
-
offerFrame({ type: "frame", seq: f.seq, data: f.data, metadata: f.metadata });
|
|
323
|
-
}
|
|
324
244
|
res.on("close", () => {
|
|
325
|
-
closed = true;
|
|
326
245
|
clearInterval(heartbeat);
|
|
327
|
-
if (slot) clearTimeout(slot);
|
|
328
|
-
unsubscribe();
|
|
329
246
|
stream.close();
|
|
330
247
|
});
|
|
331
248
|
}
|
|
@@ -338,12 +255,10 @@ module.exports = {
|
|
|
338
255
|
cdpEndpoint,
|
|
339
256
|
clampFps,
|
|
340
257
|
describeSessions,
|
|
341
|
-
driveFeed,
|
|
342
258
|
listSessions,
|
|
343
259
|
openStream,
|
|
344
260
|
processAlive,
|
|
345
261
|
safePageAddress,
|
|
346
262
|
serveStream,
|
|
347
|
-
sessionTargetId,
|
|
348
263
|
socketDirs,
|
|
349
264
|
};
|