turno-browser 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/README.md +8 -4
- package/dist/agent-cursor-overlay.d.ts +85 -0
- package/dist/agent-cursor-overlay.js +313 -0
- package/dist/agent-cursor.d.ts +159 -0
- package/dist/agent-cursor.js +408 -0
- package/dist/api.d.ts +8 -3
- package/dist/api.js +6 -3
- package/dist/argv.d.ts +2 -1
- package/dist/argv.js +7 -1
- package/dist/bin/mcp.js +5 -0
- package/dist/cli/commands.js +12 -3
- package/dist/cli/completion.js +3 -0
- package/dist/cli/context.d.ts +2 -0
- package/dist/cli/context.js +8 -4
- package/dist/cli/do-background.d.ts +2 -0
- package/dist/cli/do-background.js +15 -0
- package/dist/cli/do.d.ts +4 -0
- package/dist/cli/do.js +17 -1
- package/dist/cli/identity.d.ts +2 -0
- package/dist/cli/identity.js +64 -7
- package/dist/cli/init.d.ts +12 -0
- package/dist/cli/init.js +217 -0
- package/dist/cli/login-summary.d.ts +6 -0
- package/dist/cli/login-summary.js +11 -0
- package/dist/cli/main.js +3 -0
- package/dist/cli/setup.d.ts +32 -0
- package/dist/cli/setup.js +32 -23
- package/dist/doctor.js +18 -2
- package/dist/host.js +29 -4
- package/dist/human-input.d.ts +5 -0
- package/dist/human-input.js +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/dist/login.d.ts +22 -2
- package/dist/login.js +95 -7
- package/dist/settings.d.ts +12 -2
- package/dist/settings.js +14 -2
- package/dist/tuning.d.ts +4 -1
- package/dist/tuning.js +1 -0
- package/npm-shrinkwrap.json +2 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,32 @@
|
|
|
1
1
|
# turno-browser
|
|
2
2
|
|
|
3
|
+
## 0.7.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 69f67a1: The agent cursor: when someone can see the browser (a Chrome window, or the live view), each page shows that an agent is at work and where. A Figma-like arrow with the agent's name and the identity glides to each element it acts on, a ring marks the element, a click leaves a ripple, a field being typed into gets a typing bubble, and a thin frame with a badge ("turno · Claude Code at work", or "paused · you have the controls" while the live view's pause is on) goes around the page while calls run. Several agent sessions on one identity's Chrome each get a cursor and a color of their own, and with humanize on the arrow follows the real pointer path.
|
|
8
|
+
|
|
9
|
+
It is drawn in a closed shadow root that is out of the accessibility tree, out of the page's text and never a click target, and it is hidden while the model's screenshot is taken, so the model sees the page as before. `turno-browser set <id> --cursor auto|on|off` (identity.json `cursor`, format 12) and `turno-browser-mcp --cursor` choose it: `auto` (the default) draws it when the browser can be seen, and `off` leaves pages exactly as they are.
|
|
10
|
+
- 0c723fc: `turno-browser init` sets turno-browser up step by step: it names and creates an identity (or
|
|
11
|
+
uses one that is there), opens Chrome to sign in by hand on a first site, finds the agents
|
|
12
|
+
installed here and runs the agent's `mcp add` command only on a yes. Without a terminal it needs
|
|
13
|
+
`--identity` and `--agent`, skips the sign-in and prints the command instead. `turno-browser`
|
|
14
|
+
with no command now points to it.
|
|
15
|
+
|
|
16
|
+
The first steps say more of what happened:
|
|
17
|
+
|
|
18
|
+
- `login` opens a page that says what to do instead of a blank tab, and on close names the sites
|
|
19
|
+
that were opened, or says that nothing new was signed in.
|
|
20
|
+
- `show` has a "last sign-in" line, local times and a hint to turn on the health check;
|
|
21
|
+
`doctor` says when an identity was never signed in by hand.
|
|
22
|
+
- `list`, `show` and `doctor` say "not in use" instead of "free".
|
|
23
|
+
- `do <id> end` says that Chrome stays open in the background server and how to close it now.
|
|
24
|
+
- `do` errors no longer start with "Error: Error:", and a stale ref or text where a ref goes gets
|
|
25
|
+
a hint (`snapshot`, `find`).
|
|
26
|
+
- `setup claude` suggests `claude-code`.
|
|
27
|
+
- `add` prints its next steps with each explanation on its own line, so the commands paste into
|
|
28
|
+
zsh as they are.
|
|
29
|
+
|
|
3
30
|
## 0.6.0
|
|
4
31
|
|
|
5
32
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -11,11 +11,13 @@ that same signed-in browser afterwards.
|
|
|
11
11
|
|
|
12
12
|
```sh
|
|
13
13
|
npm install --global turno-browser
|
|
14
|
-
turno-browser
|
|
15
|
-
turno-browser login school # sign in by hand once; cookies persist
|
|
16
|
-
turno-browser setup claude-code --identity school # prints how Claude Code gets it
|
|
14
|
+
turno-browser init
|
|
17
15
|
```
|
|
18
16
|
|
|
17
|
+
`init` asks a name for the identity, opens Chrome so you sign in by hand once, and gives the
|
|
18
|
+
identity to your agent. Each step is also a command of its own: `turno-browser add`, `login` and
|
|
19
|
+
`setup`.
|
|
20
|
+
|
|
19
21
|
The docs are at [turno.wimo.dev/docs/browser](https://turno.wimo.dev/docs/browser), starting with
|
|
20
22
|
[Get started](https://turno.wimo.dev/docs/browser/get-started).
|
|
21
23
|
|
|
@@ -50,6 +52,7 @@ On from the start:
|
|
|
50
52
|
| [`browser_batch`](https://turno.wimo.dev/docs/browser/actions) | up to 20 actions in one call, and errors that say what is in the way |
|
|
51
53
|
| [Flows](https://turno.wimo.dev/docs/browser/flows) | save what an agent did on a site once, and replay it with parameters |
|
|
52
54
|
| [From the shell](https://turno.wimo.dev/docs/browser/actions#from-the-shell) | `turno-browser do <id> open`, `snapshot`, `click`, `type`, `read`: the same browser and rules, as shell commands |
|
|
55
|
+
| [The agent cursor](https://turno.wimo.dev/docs/browser/agent-cursor) | when you can see the browser, a named cursor and a badge show where the agent is working; the model never sees it |
|
|
53
56
|
| [Health, reputation and doctor](https://turno.wimo.dev/docs/browser/identities#session-health) | a signed-out identity says so, each site's challenges are counted, and `doctor` checks Chrome and every identity |
|
|
54
57
|
|
|
55
58
|
Off until you turn it on, per identity:
|
|
@@ -146,7 +149,8 @@ started it exits, Chrome closes too.
|
|
|
146
149
|
## Move a session
|
|
147
150
|
|
|
148
151
|
```sh
|
|
149
|
-
|
|
152
|
+
# asks for a passphrase twice
|
|
153
|
+
turno-browser export school --out school.session --expires 2d
|
|
150
154
|
turno-browser import school.session --as school-laptop
|
|
151
155
|
```
|
|
152
156
|
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The agent cursor's page side: one function the host evaluates in a page with a batch of
|
|
3
|
+
* commands (`agent-cursor.ts`). It runs in Playwright's utility world (patchright's
|
|
4
|
+
* `page.evaluate` default), so the state it keeps on `window` is out of the page's own scripts'
|
|
5
|
+
* reach, and it must stay self-contained: it is sent to the page as source, without imports.
|
|
6
|
+
*
|
|
7
|
+
* What it draws lives in a closed shadow root on one host element under `<html>` (never in
|
|
8
|
+
* `<body>`, so `document.body.innerText` never holds it). The host is `aria-hidden` and `inert`
|
|
9
|
+
* (out of the accessibility tree, so out of every snapshot), `pointer-events: none` (never a
|
|
10
|
+
* target of a click or of `elementFromPoint`, so Playwright's hit test and "covered" check see
|
|
11
|
+
* through it) and `position: fixed` with `contain: strict` (no layout, no scroll of its own).
|
|
12
|
+
* It builds nodes with `createElement` and styles them through the CSSOM, so a page's Trusted
|
|
13
|
+
* Types or style CSP does not stop it.
|
|
14
|
+
*/
|
|
15
|
+
/** A rectangle in viewport CSS pixels. */
|
|
16
|
+
export interface OverlayBox {
|
|
17
|
+
x: number;
|
|
18
|
+
y: number;
|
|
19
|
+
width: number;
|
|
20
|
+
height: number;
|
|
21
|
+
}
|
|
22
|
+
/** One instruction for the overlay. `key` names a cursor (one per agent session). */
|
|
23
|
+
export type OverlayCommand = {
|
|
24
|
+
op: "actor";
|
|
25
|
+
key: string;
|
|
26
|
+
label: string;
|
|
27
|
+
name: string;
|
|
28
|
+
color: string;
|
|
29
|
+
} | {
|
|
30
|
+
op: "move";
|
|
31
|
+
key: string;
|
|
32
|
+
x: number;
|
|
33
|
+
y: number;
|
|
34
|
+
ms: number;
|
|
35
|
+
} | {
|
|
36
|
+
op: "ring";
|
|
37
|
+
key: string;
|
|
38
|
+
box: OverlayBox | null;
|
|
39
|
+
} | {
|
|
40
|
+
op: "ripple";
|
|
41
|
+
key: string;
|
|
42
|
+
x: number;
|
|
43
|
+
y: number;
|
|
44
|
+
} | {
|
|
45
|
+
op: "typing";
|
|
46
|
+
key: string;
|
|
47
|
+
box: OverlayBox | null;
|
|
48
|
+
} | {
|
|
49
|
+
op: "busy";
|
|
50
|
+
key: string;
|
|
51
|
+
on: boolean;
|
|
52
|
+
} | {
|
|
53
|
+
op: "paused";
|
|
54
|
+
on: boolean;
|
|
55
|
+
} | {
|
|
56
|
+
op: "visible";
|
|
57
|
+
on: boolean;
|
|
58
|
+
} | {
|
|
59
|
+
op: "remove";
|
|
60
|
+
};
|
|
61
|
+
/** What a batch reports back: the motion preference and the viewport, for the host's glide. */
|
|
62
|
+
export interface OverlayReport {
|
|
63
|
+
reducedMotion: boolean;
|
|
64
|
+
width: number;
|
|
65
|
+
height: number;
|
|
66
|
+
/** What the overlay shows now; tests read it, since its DOM is behind a closed shadow root. */
|
|
67
|
+
state: {
|
|
68
|
+
connected: boolean;
|
|
69
|
+
visible: boolean;
|
|
70
|
+
busy: boolean;
|
|
71
|
+
paused: boolean;
|
|
72
|
+
badge: string;
|
|
73
|
+
cursors: {
|
|
74
|
+
key: string;
|
|
75
|
+
label: string;
|
|
76
|
+
x: number;
|
|
77
|
+
y: number;
|
|
78
|
+
}[];
|
|
79
|
+
};
|
|
80
|
+
}
|
|
81
|
+
/**
|
|
82
|
+
* Applies `commands` to the page's overlay, building it first when the page has none (a new
|
|
83
|
+
* document after a navigation, or a host the page removed). Runs in the page.
|
|
84
|
+
*/
|
|
85
|
+
export declare function cursorOverlay(commands: OverlayCommand[]): OverlayReport;
|
|
@@ -0,0 +1,313 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The agent cursor's page side: one function the host evaluates in a page with a batch of
|
|
3
|
+
* commands (`agent-cursor.ts`). It runs in Playwright's utility world (patchright's
|
|
4
|
+
* `page.evaluate` default), so the state it keeps on `window` is out of the page's own scripts'
|
|
5
|
+
* reach, and it must stay self-contained: it is sent to the page as source, without imports.
|
|
6
|
+
*
|
|
7
|
+
* What it draws lives in a closed shadow root on one host element under `<html>` (never in
|
|
8
|
+
* `<body>`, so `document.body.innerText` never holds it). The host is `aria-hidden` and `inert`
|
|
9
|
+
* (out of the accessibility tree, so out of every snapshot), `pointer-events: none` (never a
|
|
10
|
+
* target of a click or of `elementFromPoint`, so Playwright's hit test and "covered" check see
|
|
11
|
+
* through it) and `position: fixed` with `contain: strict` (no layout, no scroll of its own).
|
|
12
|
+
* It builds nodes with `createElement` and styles them through the CSSOM, so a page's Trusted
|
|
13
|
+
* Types or style CSP does not stop it.
|
|
14
|
+
*/
|
|
15
|
+
/**
|
|
16
|
+
* Applies `commands` to the page's overlay, building it first when the page has none (a new
|
|
17
|
+
* document after a navigation, or a host the page removed). Runs in the page.
|
|
18
|
+
*/
|
|
19
|
+
export function cursorOverlay(commands) {
|
|
20
|
+
// The global the overlay keeps its parts under, in the utility world only.
|
|
21
|
+
const GLOBAL = "__turnoAgentCursor";
|
|
22
|
+
const w = window;
|
|
23
|
+
const important = (el, props) => {
|
|
24
|
+
for (const [k, v] of Object.entries(props))
|
|
25
|
+
el.style.setProperty(k, v, "important");
|
|
26
|
+
};
|
|
27
|
+
const CSS = `
|
|
28
|
+
*{box-sizing:border-box}
|
|
29
|
+
.shell{position:fixed;inset:0;pointer-events:none;font:600 12px/16px system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;letter-spacing:.01em}
|
|
30
|
+
.frame{position:fixed;inset:0;border:2px solid var(--c);border-radius:2px;box-shadow:inset 0 0 0 1px rgba(255,255,255,.35),inset 0 0 28px color-mix(in srgb,var(--c) 22%,transparent);opacity:0;transition:opacity .35s ease}
|
|
31
|
+
.badge{position:fixed;top:10px;left:50%;display:flex;align-items:center;gap:7px;padding:5px 11px 5px 9px;border-radius:999px;background:var(--c);color:#fff;box-shadow:0 4px 14px rgba(0,0,0,.22),0 0 0 1px rgba(255,255,255,.25);white-space:nowrap;opacity:0;transform:translate(-50%,-8px);transition:opacity .35s ease,transform .35s ease}
|
|
32
|
+
.dot{width:7px;height:7px;border-radius:50%;background:#fff;box-shadow:0 0 0 0 rgba(255,255,255,.7);animation:pulse 1.4s ease-out infinite}
|
|
33
|
+
.on .frame{opacity:1}
|
|
34
|
+
.on .badge{opacity:1;transform:translate(-50%,0)}
|
|
35
|
+
.paused .frame{border-style:dashed}
|
|
36
|
+
.paused .dot{animation:none;border-radius:1px;width:6px;height:8px;background:linear-gradient(90deg,#fff 0 35%,transparent 35% 65%,#fff 65%)}
|
|
37
|
+
.cursor{position:fixed;left:0;top:0;transform:translate(var(--x),var(--y));transition:transform var(--ms) cubic-bezier(.22,.61,.36,1),opacity .4s ease;will-change:transform}
|
|
38
|
+
.cursor svg{display:block;overflow:visible;filter:drop-shadow(0 2px 3px rgba(0,0,0,.3))}
|
|
39
|
+
.pill{position:absolute;left:15px;top:19px;padding:3px 8px;border-radius:3px 10px 10px 10px;background:var(--c);color:#fff;white-space:nowrap;box-shadow:0 2px 8px rgba(0,0,0,.2)}
|
|
40
|
+
.idle .cursor{opacity:.6}
|
|
41
|
+
.ring{position:fixed;border:2px solid var(--c);border-radius:7px;box-shadow:0 0 0 4px color-mix(in srgb,var(--c) 20%,transparent);opacity:0;transition:opacity .25s ease,left .2s ease,top .2s ease,width .2s ease,height .2s ease}
|
|
42
|
+
.ring.show{opacity:1}
|
|
43
|
+
.ripple{position:fixed;width:14px;height:14px;margin:-7px 0 0 -7px;border-radius:50%;border:2px solid var(--c);background:color-mix(in srgb,var(--c) 25%,transparent);animation:ripple .6s ease-out forwards}
|
|
44
|
+
.typing{position:fixed;display:flex;align-items:center;gap:3px;padding:4px 7px;border-radius:10px;background:var(--c);color:#fff;box-shadow:0 2px 8px rgba(0,0,0,.2)}
|
|
45
|
+
.typing i{width:4px;height:4px;border-radius:50%;background:#fff;animation:blink 1s infinite}
|
|
46
|
+
.typing i:nth-child(2){animation-delay:.15s}
|
|
47
|
+
.typing i:nth-child(3){animation-delay:.3s}
|
|
48
|
+
@keyframes pulse{0%{box-shadow:0 0 0 0 rgba(255,255,255,.7)}70%{box-shadow:0 0 0 6px rgba(255,255,255,0)}100%{box-shadow:0 0 0 0 rgba(255,255,255,0)}}
|
|
49
|
+
@keyframes ripple{to{transform:scale(3.4);opacity:0}}
|
|
50
|
+
@keyframes blink{0%,100%{opacity:.35}50%{opacity:1}}
|
|
51
|
+
@media (prefers-reduced-motion: reduce){
|
|
52
|
+
.frame,.badge,.cursor,.ring{transition:none}
|
|
53
|
+
.dot,.typing i{animation:none}
|
|
54
|
+
.ripple{animation-duration:.01s}
|
|
55
|
+
}`;
|
|
56
|
+
const build = () => {
|
|
57
|
+
const host = document.createElement("turno-agent-cursor");
|
|
58
|
+
host.setAttribute("aria-hidden", "true");
|
|
59
|
+
host.setAttribute("inert", "");
|
|
60
|
+
important(host, {
|
|
61
|
+
all: "initial",
|
|
62
|
+
display: "block",
|
|
63
|
+
position: "fixed",
|
|
64
|
+
inset: "0",
|
|
65
|
+
width: "100vw",
|
|
66
|
+
height: "100vh",
|
|
67
|
+
margin: "0",
|
|
68
|
+
padding: "0",
|
|
69
|
+
border: "0",
|
|
70
|
+
"z-index": "2147483647",
|
|
71
|
+
"pointer-events": "none",
|
|
72
|
+
contain: "strict",
|
|
73
|
+
overflow: "hidden",
|
|
74
|
+
background: "transparent",
|
|
75
|
+
});
|
|
76
|
+
const root = host.attachShadow({ mode: "closed" });
|
|
77
|
+
try {
|
|
78
|
+
const sheet = new CSSStyleSheet();
|
|
79
|
+
sheet.replaceSync(CSS);
|
|
80
|
+
root.adoptedStyleSheets = [sheet];
|
|
81
|
+
}
|
|
82
|
+
catch {
|
|
83
|
+
const style = document.createElement("style");
|
|
84
|
+
style.textContent = CSS;
|
|
85
|
+
root.append(style);
|
|
86
|
+
}
|
|
87
|
+
const shell = document.createElement("div");
|
|
88
|
+
shell.className = "shell idle";
|
|
89
|
+
const frame = document.createElement("div");
|
|
90
|
+
frame.className = "frame";
|
|
91
|
+
const badge = document.createElement("div");
|
|
92
|
+
badge.className = "badge";
|
|
93
|
+
const dot = document.createElement("span");
|
|
94
|
+
dot.className = "dot";
|
|
95
|
+
const badgeText = document.createElement("span");
|
|
96
|
+
badgeText.textContent = "turno · agent at work";
|
|
97
|
+
badge.append(dot, badgeText);
|
|
98
|
+
const layer = document.createElement("div");
|
|
99
|
+
shell.append(frame, layer, badge);
|
|
100
|
+
root.append(shell);
|
|
101
|
+
document.documentElement.append(host);
|
|
102
|
+
const reduced = window.matchMedia?.("(prefers-reduced-motion: reduce)").matches ?? false;
|
|
103
|
+
return {
|
|
104
|
+
doc: document,
|
|
105
|
+
host,
|
|
106
|
+
root,
|
|
107
|
+
shell,
|
|
108
|
+
frame,
|
|
109
|
+
badge,
|
|
110
|
+
badgeText,
|
|
111
|
+
layer,
|
|
112
|
+
cursors: new Map(),
|
|
113
|
+
rings: new Map(),
|
|
114
|
+
typing: new Map(),
|
|
115
|
+
timers: new Map(),
|
|
116
|
+
busy: new Set(),
|
|
117
|
+
lastBusy: "",
|
|
118
|
+
paused: false,
|
|
119
|
+
visible: true,
|
|
120
|
+
reduced,
|
|
121
|
+
};
|
|
122
|
+
};
|
|
123
|
+
let p = w[GLOBAL];
|
|
124
|
+
if (!p || p.doc !== document) {
|
|
125
|
+
p = build();
|
|
126
|
+
w[GLOBAL] = p;
|
|
127
|
+
}
|
|
128
|
+
else if (!p.host.isConnected) {
|
|
129
|
+
// The page replaced <html>'s children (or the host itself): put it back.
|
|
130
|
+
document.documentElement.append(p.host);
|
|
131
|
+
}
|
|
132
|
+
const parts = p;
|
|
133
|
+
const later = (name, ms, fn) => {
|
|
134
|
+
const old = parts.timers.get(name);
|
|
135
|
+
if (old !== undefined)
|
|
136
|
+
clearTimeout(old);
|
|
137
|
+
parts.timers.set(name, window.setTimeout(() => {
|
|
138
|
+
parts.timers.delete(name);
|
|
139
|
+
fn();
|
|
140
|
+
}, ms));
|
|
141
|
+
};
|
|
142
|
+
const cursorFor = (key) => {
|
|
143
|
+
let c = parts.cursors.get(key);
|
|
144
|
+
if (c)
|
|
145
|
+
return c;
|
|
146
|
+
const el = document.createElement("div");
|
|
147
|
+
el.className = "cursor";
|
|
148
|
+
el.style.setProperty("--c", "#2b59ff");
|
|
149
|
+
el.style.setProperty("--x", "-40px");
|
|
150
|
+
el.style.setProperty("--y", "-40px");
|
|
151
|
+
el.style.setProperty("--ms", "0ms");
|
|
152
|
+
const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
|
|
153
|
+
svg.setAttribute("width", "22");
|
|
154
|
+
svg.setAttribute("height", "26");
|
|
155
|
+
svg.setAttribute("viewBox", "0 0 22 26");
|
|
156
|
+
const path = document.createElementNS("http://www.w3.org/2000/svg", "path");
|
|
157
|
+
// A Figma-like arrow, its tip at (2, 2).
|
|
158
|
+
path.setAttribute("d", "M2 2 L2 21 L7.2 16.3 L10.6 23.6 L14 22.1 L10.6 14.9 L17.6 14.9 Z");
|
|
159
|
+
path.setAttribute("fill", "var(--c)");
|
|
160
|
+
path.setAttribute("stroke", "#fff");
|
|
161
|
+
path.setAttribute("stroke-width", "1.6");
|
|
162
|
+
path.setAttribute("stroke-linejoin", "round");
|
|
163
|
+
svg.append(path);
|
|
164
|
+
const pill = document.createElement("div");
|
|
165
|
+
pill.className = "pill";
|
|
166
|
+
el.append(svg, pill);
|
|
167
|
+
parts.layer.append(el);
|
|
168
|
+
c = { el, pill, label: "", name: "", color: "#2b59ff", x: -40, y: -40 };
|
|
169
|
+
parts.cursors.set(key, c);
|
|
170
|
+
return c;
|
|
171
|
+
};
|
|
172
|
+
const colorOf = (key) => parts.cursors.get(key)?.color ?? "#2b59ff";
|
|
173
|
+
const paintBadge = () => {
|
|
174
|
+
const on = parts.busy.size > 0 || parts.paused;
|
|
175
|
+
parts.shell.classList.toggle("on", on);
|
|
176
|
+
parts.shell.classList.toggle("idle", !on);
|
|
177
|
+
parts.shell.classList.toggle("paused", parts.paused);
|
|
178
|
+
const key = parts.busy.has(parts.lastBusy) ? parts.lastBusy : ([...parts.busy][0] ?? parts.lastBusy);
|
|
179
|
+
const color = parts.paused ? "#5c5f66" : colorOf(key);
|
|
180
|
+
parts.frame.style.setProperty("--c", color);
|
|
181
|
+
parts.badge.style.setProperty("--c", color);
|
|
182
|
+
const name = parts.cursors.get(key)?.name;
|
|
183
|
+
const others = parts.busy.size > 1 ? ` +${parts.busy.size - 1}` : "";
|
|
184
|
+
parts.badgeText.textContent = parts.paused
|
|
185
|
+
? "paused · you have the controls"
|
|
186
|
+
: `turno · ${name ? `${name} at work` : "agent at work"}${others}`;
|
|
187
|
+
};
|
|
188
|
+
for (const cmd of commands) {
|
|
189
|
+
switch (cmd.op) {
|
|
190
|
+
case "actor": {
|
|
191
|
+
const c = cursorFor(cmd.key);
|
|
192
|
+
c.label = cmd.label;
|
|
193
|
+
c.name = cmd.name;
|
|
194
|
+
c.color = cmd.color;
|
|
195
|
+
c.pill.textContent = cmd.label;
|
|
196
|
+
c.el.style.setProperty("--c", cmd.color);
|
|
197
|
+
break;
|
|
198
|
+
}
|
|
199
|
+
case "move": {
|
|
200
|
+
const c = cursorFor(cmd.key);
|
|
201
|
+
c.x = cmd.x;
|
|
202
|
+
c.y = cmd.y;
|
|
203
|
+
c.el.style.setProperty("--ms", `${parts.reduced ? 0 : Math.max(0, Math.round(cmd.ms))}ms`);
|
|
204
|
+
// The arrow's tip is 2px into its box.
|
|
205
|
+
c.el.style.setProperty("--x", `${Math.round(cmd.x - 2)}px`);
|
|
206
|
+
c.el.style.setProperty("--y", `${Math.round(cmd.y - 2)}px`);
|
|
207
|
+
break;
|
|
208
|
+
}
|
|
209
|
+
case "ring": {
|
|
210
|
+
let ring = parts.rings.get(cmd.key);
|
|
211
|
+
if (!cmd.box) {
|
|
212
|
+
ring?.classList.remove("show");
|
|
213
|
+
break;
|
|
214
|
+
}
|
|
215
|
+
if (!ring) {
|
|
216
|
+
ring = document.createElement("div");
|
|
217
|
+
ring.className = "ring";
|
|
218
|
+
parts.layer.prepend(ring);
|
|
219
|
+
parts.rings.set(cmd.key, ring);
|
|
220
|
+
}
|
|
221
|
+
const pad = 4;
|
|
222
|
+
ring.style.setProperty("--c", colorOf(cmd.key));
|
|
223
|
+
ring.style.left = `${Math.round(cmd.box.x - pad)}px`;
|
|
224
|
+
ring.style.top = `${Math.round(cmd.box.y - pad)}px`;
|
|
225
|
+
ring.style.width = `${Math.round(cmd.box.width + 2 * pad)}px`;
|
|
226
|
+
ring.style.height = `${Math.round(cmd.box.height + 2 * pad)}px`;
|
|
227
|
+
ring.classList.add("show");
|
|
228
|
+
const r = ring;
|
|
229
|
+
later(`ring:${cmd.key}`, 1600, () => r.classList.remove("show"));
|
|
230
|
+
break;
|
|
231
|
+
}
|
|
232
|
+
case "ripple": {
|
|
233
|
+
const dot = document.createElement("div");
|
|
234
|
+
dot.className = "ripple";
|
|
235
|
+
dot.style.setProperty("--c", colorOf(cmd.key));
|
|
236
|
+
dot.style.left = `${Math.round(cmd.x)}px`;
|
|
237
|
+
dot.style.top = `${Math.round(cmd.y)}px`;
|
|
238
|
+
parts.layer.append(dot);
|
|
239
|
+
window.setTimeout(() => dot.remove(), 700);
|
|
240
|
+
break;
|
|
241
|
+
}
|
|
242
|
+
case "typing": {
|
|
243
|
+
const old = parts.typing.get(cmd.key);
|
|
244
|
+
if (!cmd.box) {
|
|
245
|
+
old?.remove();
|
|
246
|
+
parts.typing.delete(cmd.key);
|
|
247
|
+
break;
|
|
248
|
+
}
|
|
249
|
+
const bubble = old ?? document.createElement("div");
|
|
250
|
+
if (!old) {
|
|
251
|
+
bubble.className = "typing";
|
|
252
|
+
bubble.append(document.createElement("i"), document.createElement("i"), document.createElement("i"));
|
|
253
|
+
parts.layer.append(bubble);
|
|
254
|
+
parts.typing.set(cmd.key, bubble);
|
|
255
|
+
}
|
|
256
|
+
bubble.style.setProperty("--c", colorOf(cmd.key));
|
|
257
|
+
const right = cmd.box.x + cmd.box.width;
|
|
258
|
+
bubble.style.left = `${Math.round(Math.min(right + 6, window.innerWidth - 40))}px`;
|
|
259
|
+
bubble.style.top = `${Math.round(Math.max(cmd.box.y + cmd.box.height / 2 - 9, 2))}px`;
|
|
260
|
+
break;
|
|
261
|
+
}
|
|
262
|
+
case "busy": {
|
|
263
|
+
if (cmd.on) {
|
|
264
|
+
parts.busy.add(cmd.key);
|
|
265
|
+
parts.lastBusy = cmd.key;
|
|
266
|
+
const t = parts.timers.get(`idle:${cmd.key}`);
|
|
267
|
+
if (t !== undefined) {
|
|
268
|
+
clearTimeout(t);
|
|
269
|
+
parts.timers.delete(`idle:${cmd.key}`);
|
|
270
|
+
}
|
|
271
|
+
paintBadge();
|
|
272
|
+
}
|
|
273
|
+
else {
|
|
274
|
+
// The frame stays a moment after a call, so a run of calls reads as one stretch of work.
|
|
275
|
+
later(`idle:${cmd.key}`, 2500, () => {
|
|
276
|
+
parts.busy.delete(cmd.key);
|
|
277
|
+
paintBadge();
|
|
278
|
+
});
|
|
279
|
+
}
|
|
280
|
+
break;
|
|
281
|
+
}
|
|
282
|
+
case "paused":
|
|
283
|
+
parts.paused = cmd.on;
|
|
284
|
+
paintBadge();
|
|
285
|
+
break;
|
|
286
|
+
case "visible":
|
|
287
|
+
parts.visible = cmd.on;
|
|
288
|
+
parts.host.style.setProperty("visibility", cmd.on ? "visible" : "hidden", "important");
|
|
289
|
+
break;
|
|
290
|
+
case "remove":
|
|
291
|
+
for (const t of parts.timers.values())
|
|
292
|
+
clearTimeout(t);
|
|
293
|
+
parts.host.remove();
|
|
294
|
+
w[GLOBAL] = undefined;
|
|
295
|
+
break;
|
|
296
|
+
}
|
|
297
|
+
}
|
|
298
|
+
const current = w[GLOBAL];
|
|
299
|
+
return {
|
|
300
|
+
reducedMotion: parts.reduced,
|
|
301
|
+
width: window.innerWidth,
|
|
302
|
+
height: window.innerHeight,
|
|
303
|
+
state: {
|
|
304
|
+
connected: current !== undefined && parts.host.isConnected,
|
|
305
|
+
visible: parts.visible,
|
|
306
|
+
busy: parts.busy.size > 0,
|
|
307
|
+
paused: parts.paused,
|
|
308
|
+
badge: parts.badgeText.textContent ?? "",
|
|
309
|
+
cursors: [...parts.cursors.entries()].map(([key, c]) => ({ key, label: c.label, x: c.x, y: c.y })),
|
|
310
|
+
},
|
|
311
|
+
};
|
|
312
|
+
}
|
|
313
|
+
//# sourceMappingURL=agent-cursor-overlay.js.map
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
import type { Locator, Page } from "patchright-core";
|
|
2
|
+
import { type OverlayBox, type OverlayCommand, type OverlayReport } from "./agent-cursor-overlay.js";
|
|
3
|
+
import type { CursorMode } from "./settings.js";
|
|
4
|
+
import type { UpstreamTool } from "./upstream.js";
|
|
5
|
+
/**
|
|
6
|
+
* The agent cursor: while an agent drives a browser someone can see (a headed Chrome window, or
|
|
7
|
+
* any browser with the live view on), each page shows who is at work and where. A Figma-like
|
|
8
|
+
* arrow with the agent's name glides to each element it acts on, a ring marks the element, a
|
|
9
|
+
* click leaves a ripple, a field being typed into gets a typing bubble, and a thin frame with a
|
|
10
|
+
* badge ("turno · Claude Code at work", "paused · you have the controls") goes around the page
|
|
11
|
+
* while calls run. Several agent sessions on one identity's Chrome (`"sharing": "tabs"`) each
|
|
12
|
+
* get a cursor and a color of their own.
|
|
13
|
+
*
|
|
14
|
+
* The coordinates are the agent's own: the input tools run on a view of the tab (`CursorInput`)
|
|
15
|
+
* that reads the target's box before each action, as `human-input.ts` does for human-like input.
|
|
16
|
+
* Nothing follows the real mouse, so the user's own pointer never moves the agent's cursor.
|
|
17
|
+
* With humanize on, its pointer path goes through the same view, so the arrow follows the curve.
|
|
18
|
+
*
|
|
19
|
+
* What the page draws is the overlay of `agent-cursor-overlay.ts`, built on first use in each
|
|
20
|
+
* document and kept out of the model's view: out of the accessibility tree (snapshots), out of
|
|
21
|
+
* `document.body` (read), never a hit target, and hidden while the model's screenshot is taken
|
|
22
|
+
* (`hiddenDuring`). Recordings and the live view keep it: they are for the person.
|
|
23
|
+
*/
|
|
24
|
+
/** What the policy needs to know about the browser being launched. */
|
|
25
|
+
export interface CursorContext {
|
|
26
|
+
/** Chrome runs without a window. */
|
|
27
|
+
headless: boolean;
|
|
28
|
+
/** The live view is on for the identity. */
|
|
29
|
+
live: boolean;
|
|
30
|
+
/** The engine that runs it; read mode lays out no page to draw on. */
|
|
31
|
+
engine: string;
|
|
32
|
+
}
|
|
33
|
+
/**
|
|
34
|
+
* Whether the agent cursor is drawn: never in read mode, always with `on`, and with `auto` (the
|
|
35
|
+
* default) when someone can see the browser: a Chrome window, or the live view.
|
|
36
|
+
*/
|
|
37
|
+
export declare function resolveCursor(mode: CursorMode | undefined, ctx: CursorContext): boolean;
|
|
38
|
+
/** One color per agent session, in order: turno's blue first. */
|
|
39
|
+
export declare const CURSOR_COLORS: readonly ["#2b59ff", "#e8590c", "#2f9e44", "#ae3ec9", "#d6336c", "#0c8599"];
|
|
40
|
+
/** The longest glide to a target: long enough to follow, short enough to cost little. */
|
|
41
|
+
export declare const MAX_GLIDE_MS = 420;
|
|
42
|
+
export interface Point {
|
|
43
|
+
x: number;
|
|
44
|
+
y: number;
|
|
45
|
+
}
|
|
46
|
+
/** How long the arrow takes from `from` to `to`: a short base plus the distance, capped. */
|
|
47
|
+
export declare function glideMs(from: Point | undefined, to: Point, max?: number): number;
|
|
48
|
+
/** The center of a box. */
|
|
49
|
+
export declare function centerOf(box: OverlayBox): Point;
|
|
50
|
+
/** True when some of the box is inside a viewport of this size. */
|
|
51
|
+
export declare function inView(box: OverlayBox, view: {
|
|
52
|
+
width: number;
|
|
53
|
+
height: number;
|
|
54
|
+
}): boolean;
|
|
55
|
+
/** The pill of a session's cursor: the client's name, and the identity when there is one. */
|
|
56
|
+
export declare function cursorLabel(client: string, identity?: string): string;
|
|
57
|
+
interface Actor {
|
|
58
|
+
key: string;
|
|
59
|
+
/** The client's name: the badge says it is at work. */
|
|
60
|
+
name: string;
|
|
61
|
+
/** The cursor's pill: the name and the identity. */
|
|
62
|
+
label: string;
|
|
63
|
+
color: string;
|
|
64
|
+
}
|
|
65
|
+
export interface AgentCursorOptions {
|
|
66
|
+
/** The client name of the session that launched the browser (its window title). */
|
|
67
|
+
title: string;
|
|
68
|
+
identity?: string;
|
|
69
|
+
sleep?: (ms: number) => Promise<void>;
|
|
70
|
+
/** The longest glide; 0 places the arrow without a glide. */
|
|
71
|
+
maxGlideMs?: number;
|
|
72
|
+
}
|
|
73
|
+
/** The agent cursor of one running browser: its actors and what each page shows. */
|
|
74
|
+
export declare class AgentCursor {
|
|
75
|
+
private readonly options;
|
|
76
|
+
private readonly actors;
|
|
77
|
+
/** The session acting on each page now (set when a call begins on it). */
|
|
78
|
+
private readonly owners;
|
|
79
|
+
/** Where each session's arrow is on each page, so a new document puts it back. */
|
|
80
|
+
private readonly points;
|
|
81
|
+
private readonly reports;
|
|
82
|
+
private readonly sleep;
|
|
83
|
+
private readonly maxGlideMs;
|
|
84
|
+
private lastKey;
|
|
85
|
+
private paused;
|
|
86
|
+
constructor(options: AgentCursorOptions);
|
|
87
|
+
/** The actor of a session, with its label and color, made on first use. */
|
|
88
|
+
actor(key: string): Actor;
|
|
89
|
+
/** The session acting on this page now, or the last one that acted anywhere. */
|
|
90
|
+
ownerOf(page: Page): string;
|
|
91
|
+
/** Where the session's arrow is on this page, if it has been there. */
|
|
92
|
+
pointOf(page: Page, key?: string): Point | undefined;
|
|
93
|
+
/** The viewport and motion preference the page last reported. */
|
|
94
|
+
reportOf(page: Page): OverlayReport | undefined;
|
|
95
|
+
/**
|
|
96
|
+
* Sends commands to the page's overlay, with the actor and the arrow's last place first, so a
|
|
97
|
+
* document the overlay has not been on yet shows the same thing. Never throws: a page that
|
|
98
|
+
* closed or navigated away just misses one frame.
|
|
99
|
+
*/
|
|
100
|
+
send(page: Page, commands: OverlayCommand[], key?: string): Promise<OverlayReport | undefined>;
|
|
101
|
+
private remember;
|
|
102
|
+
/** A call of session `key` begins on this page: the frame and badge come on. */
|
|
103
|
+
begin(page: Page | undefined, key: string): Promise<void>;
|
|
104
|
+
/** The call ended (on this page, which may be a new one): the frame fades after a moment. */
|
|
105
|
+
end(page: Page | undefined, key: string): Promise<void>;
|
|
106
|
+
/** Glides the arrow to `to` and waits for it to arrive. */
|
|
107
|
+
glideTo(page: Page, to: Point): Promise<void>;
|
|
108
|
+
/** Moves the arrow along with a pointer path someone else plays (humanize): a short ease per step. */
|
|
109
|
+
track(page: Page, to: Point): Promise<void>;
|
|
110
|
+
/** Puts the arrow at `to` without a glide. */
|
|
111
|
+
place(page: Page, to: Point): Promise<void>;
|
|
112
|
+
ring(page: Page, box: OverlayBox | null): Promise<void>;
|
|
113
|
+
ripple(page: Page, at: Point): Promise<void>;
|
|
114
|
+
typing(page: Page, box: OverlayBox | null): Promise<void>;
|
|
115
|
+
/** The live view's pause: every page says the user has the controls. */
|
|
116
|
+
setPaused(pages: readonly Page[], on: boolean): Promise<void>;
|
|
117
|
+
/** Runs `fn` with the page's overlay hidden: the model's screenshot shows the page alone. */
|
|
118
|
+
hiddenDuring<T>(page: Page | undefined, fn: () => Promise<T>): Promise<T>;
|
|
119
|
+
}
|
|
120
|
+
/**
|
|
121
|
+
* The input tools' view of a tab that moves the agent cursor: each locator action glides the
|
|
122
|
+
* arrow to its target first, and the coordinate tools' mouse does the same. With humanize on
|
|
123
|
+
* (`humanized`), the pointer path humanize plays goes through this view's mouse, so the arrow
|
|
124
|
+
* follows it step by step instead of gliding on its own.
|
|
125
|
+
*/
|
|
126
|
+
export declare class CursorInput {
|
|
127
|
+
readonly cursor: AgentCursor;
|
|
128
|
+
private readonly humanized;
|
|
129
|
+
/** One view per page, so humanize, which keys its pointer state by page, sees the same one. */
|
|
130
|
+
private readonly views;
|
|
131
|
+
/** The last element marked on each page: where keys typed through the keyboard go. */
|
|
132
|
+
private readonly marked;
|
|
133
|
+
private readonly typingTimers;
|
|
134
|
+
constructor(cursor: AgentCursor, humanized: boolean);
|
|
135
|
+
private boxOf;
|
|
136
|
+
/** Marks the target of an action, runs it, and leaves the mark of its kind. */
|
|
137
|
+
private act;
|
|
138
|
+
/** The cursor view of one of the page's locators. */
|
|
139
|
+
wrapLocator(locator: Locator, page: Page): Locator;
|
|
140
|
+
/** The cursor view of the page's mouse (the coordinate tools, and humanize's pointer path). */
|
|
141
|
+
private wrapMouse;
|
|
142
|
+
/** Keys typed through the keyboard (humanize's typing, browser_press_key): a typing bubble a moment. */
|
|
143
|
+
private keys;
|
|
144
|
+
/** The cursor view of the page's keyboard. */
|
|
145
|
+
private wrapKeyboard;
|
|
146
|
+
/** The cursor view of the page itself: the same object for every call on that page. */
|
|
147
|
+
private viewOf;
|
|
148
|
+
/** A view of one of upstream's tabs whose locators and mouse move the agent cursor. */
|
|
149
|
+
wrapTab<T extends object>(tab: T): T;
|
|
150
|
+
/** A view of upstream's per-session `Context` whose `ensureTab` hands out cursor tabs. */
|
|
151
|
+
wrapContext<T extends object>(context: T): T;
|
|
152
|
+
}
|
|
153
|
+
/**
|
|
154
|
+
* The stock tools of one browser, with the input tools running on cursor tabs. It goes under
|
|
155
|
+
* `humanizeTools` (applied to its result), so humanize's pointer path runs through the cursor's
|
|
156
|
+
* mouse. Schemas and every other tool are upstream's own objects.
|
|
157
|
+
*/
|
|
158
|
+
export declare function cursorTools<T extends UpstreamTool>(tools: readonly T[], input: CursorInput): T[];
|
|
159
|
+
export {};
|