libretto 0.4.4 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli/cli.js +20 -19
- package/dist/cli/commands/ai.js +1 -1
- package/dist/cli/commands/browser.js +3 -3
- package/dist/cli/commands/execution.js +3 -3
- package/dist/cli/commands/logs.js +1 -1
- package/dist/cli/core/browser.js +11 -6
- package/dist/cli/core/context.js +4 -18
- package/dist/cli/core/session.js +2 -2
- package/dist/cli/core/snapshot-analyzer.js +2 -2
- package/dist/cli/router.js +1 -1
- package/dist/cli/workers/run-integration-runtime.js +2 -2
- package/dist/shared/paths/paths.js +2 -1
- package/dist/shared/paths/repo-root.d.ts +3 -0
- package/dist/shared/paths/repo-root.js +24 -0
- package/package.json +6 -7
- package/scripts/postinstall.mjs +12 -3
- package/skills/libretto/SKILL.md +93 -404
- package/skills/libretto/references/auth-profiles.md +30 -0
- package/skills/libretto/references/pages-and-page-targeting.md +29 -0
- package/skills/libretto/references/reverse-engineering-network-requests.md +39 -0
- package/skills/libretto/references/user-action-log.md +31 -0
- package/src/cli/cli.ts +173 -0
- package/src/cli/commands/ai.ts +35 -0
- package/src/cli/commands/browser.ts +165 -0
- package/src/cli/commands/execution.ts +691 -0
- package/src/cli/commands/init.ts +327 -0
- package/src/cli/commands/logs.ts +128 -0
- package/src/cli/commands/shared.ts +70 -0
- package/src/cli/commands/snapshot.ts +327 -0
- package/src/cli/core/ai-config.ts +255 -0
- package/src/cli/core/api-snapshot-analyzer.ts +97 -0
- package/src/cli/core/browser.ts +839 -0
- package/src/cli/core/context.ts +122 -0
- package/src/cli/core/pause-signals.ts +35 -0
- package/src/cli/core/session-telemetry.ts +553 -0
- package/src/cli/core/session.ts +209 -0
- package/src/cli/core/snapshot-analyzer.ts +875 -0
- package/src/cli/core/snapshot-api-config.ts +236 -0
- package/src/cli/core/telemetry.ts +446 -0
- package/src/cli/framework/simple-cli.ts +1273 -0
- package/src/cli/index.ts +13 -0
- package/src/cli/router.ts +28 -0
- package/src/cli/workers/run-integration-runtime.ts +311 -0
- package/src/cli/workers/run-integration-worker-protocol.ts +14 -0
- package/src/cli/workers/run-integration-worker.ts +75 -0
- package/src/index.ts +120 -0
- package/src/runtime/download/download.ts +100 -0
- package/src/runtime/download/index.ts +7 -0
- package/src/runtime/extract/extract.ts +92 -0
- package/src/runtime/extract/index.ts +1 -0
- package/src/runtime/network/index.ts +5 -0
- package/src/runtime/network/network.ts +113 -0
- package/src/runtime/recovery/agent.ts +256 -0
- package/src/runtime/recovery/errors.ts +152 -0
- package/src/runtime/recovery/index.ts +7 -0
- package/src/runtime/recovery/recovery.ts +50 -0
- package/{dist/shared/condense-dom/condense-dom.cjs → src/shared/condense-dom/condense-dom.ts} +243 -115
- package/src/shared/config/config.ts +22 -0
- package/src/shared/config/index.ts +5 -0
- package/src/shared/debug/index.ts +1 -0
- package/src/shared/debug/pause.ts +85 -0
- package/src/shared/instrumentation/errors.ts +82 -0
- package/src/shared/instrumentation/index.ts +9 -0
- package/src/shared/instrumentation/instrument.ts +276 -0
- package/src/shared/llm/ai-sdk-adapter.ts +78 -0
- package/src/shared/llm/client.ts +217 -0
- package/src/shared/llm/index.ts +3 -0
- package/src/shared/llm/types.ts +63 -0
- package/src/shared/logger/index.ts +6 -0
- package/src/shared/logger/logger.ts +352 -0
- package/src/shared/logger/sinks.ts +144 -0
- package/src/shared/paths/paths.ts +109 -0
- package/src/shared/paths/repo-root.ts +27 -0
- package/src/shared/run/api.ts +2 -0
- package/src/shared/run/browser.ts +98 -0
- package/src/shared/state/index.ts +11 -0
- package/src/shared/state/session-state.ts +74 -0
- package/src/shared/visualization/ghost-cursor.ts +200 -0
- package/src/shared/visualization/highlight.ts +146 -0
- package/src/shared/visualization/index.ts +18 -0
- package/src/shared/workflow/workflow.ts +42 -0
- package/dist/index.cjs +0 -144
- package/dist/index.d.cts +0 -21
- package/dist/runtime/download/download.cjs +0 -70
- package/dist/runtime/download/download.d.cts +0 -35
- package/dist/runtime/download/index.cjs +0 -30
- package/dist/runtime/download/index.d.cts +0 -3
- package/dist/runtime/extract/extract.cjs +0 -88
- package/dist/runtime/extract/extract.d.cts +0 -23
- package/dist/runtime/extract/index.cjs +0 -28
- package/dist/runtime/extract/index.d.cts +0 -5
- package/dist/runtime/network/index.cjs +0 -28
- package/dist/runtime/network/index.d.cts +0 -4
- package/dist/runtime/network/network.cjs +0 -91
- package/dist/runtime/network/network.d.cts +0 -28
- package/dist/runtime/recovery/agent.cjs +0 -223
- package/dist/runtime/recovery/agent.d.cts +0 -13
- package/dist/runtime/recovery/errors.cjs +0 -124
- package/dist/runtime/recovery/errors.d.cts +0 -31
- package/dist/runtime/recovery/index.cjs +0 -34
- package/dist/runtime/recovery/index.d.cts +0 -7
- package/dist/runtime/recovery/recovery.cjs +0 -55
- package/dist/runtime/recovery/recovery.d.cts +0 -12
- package/dist/shared/condense-dom/condense-dom.d.cts +0 -34
- package/dist/shared/config/config.cjs +0 -44
- package/dist/shared/config/config.d.cts +0 -10
- package/dist/shared/config/index.cjs +0 -32
- package/dist/shared/config/index.d.cts +0 -1
- package/dist/shared/debug/index.cjs +0 -28
- package/dist/shared/debug/index.d.cts +0 -1
- package/dist/shared/debug/pause.cjs +0 -86
- package/dist/shared/debug/pause.d.cts +0 -12
- package/dist/shared/instrumentation/errors.cjs +0 -81
- package/dist/shared/instrumentation/errors.d.cts +0 -12
- package/dist/shared/instrumentation/index.cjs +0 -35
- package/dist/shared/instrumentation/index.d.cts +0 -6
- package/dist/shared/instrumentation/instrument.cjs +0 -206
- package/dist/shared/instrumentation/instrument.d.cts +0 -32
- package/dist/shared/llm/ai-sdk-adapter.cjs +0 -71
- package/dist/shared/llm/ai-sdk-adapter.d.cts +0 -22
- package/dist/shared/llm/client.cjs +0 -218
- package/dist/shared/llm/client.d.cts +0 -13
- package/dist/shared/llm/index.cjs +0 -31
- package/dist/shared/llm/index.d.cts +0 -5
- package/dist/shared/llm/types.cjs +0 -16
- package/dist/shared/llm/types.d.cts +0 -67
- package/dist/shared/logger/index.cjs +0 -37
- package/dist/shared/logger/index.d.cts +0 -2
- package/dist/shared/logger/logger.cjs +0 -232
- package/dist/shared/logger/logger.d.cts +0 -86
- package/dist/shared/logger/sinks.cjs +0 -160
- package/dist/shared/logger/sinks.d.cts +0 -9
- package/dist/shared/paths/paths.cjs +0 -104
- package/dist/shared/paths/paths.d.cts +0 -10
- package/dist/shared/run/api.cjs +0 -28
- package/dist/shared/run/api.d.cts +0 -2
- package/dist/shared/run/browser.cjs +0 -98
- package/dist/shared/run/browser.d.cts +0 -22
- package/dist/shared/state/index.cjs +0 -38
- package/dist/shared/state/index.d.cts +0 -2
- package/dist/shared/state/session-state.cjs +0 -92
- package/dist/shared/state/session-state.d.cts +0 -40
- package/dist/shared/visualization/ghost-cursor.cjs +0 -174
- package/dist/shared/visualization/ghost-cursor.d.cts +0 -37
- package/dist/shared/visualization/highlight.cjs +0 -134
- package/dist/shared/visualization/highlight.d.cts +0 -22
- package/dist/shared/visualization/index.cjs +0 -45
- package/dist/shared/visualization/index.d.cts +0 -3
- package/dist/shared/workflow/workflow.cjs +0 -47
- package/dist/shared/workflow/workflow.d.cts +0 -21
- package/skills/libretto/code-generation-rules.md +0 -223
- package/skills/libretto/integration-approach-selection.md +0 -174
package/skills/libretto/SKILL.md
CHANGED
|
@@ -1,453 +1,142 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: libretto
|
|
3
|
-
description: "Browser automation CLI for
|
|
3
|
+
description: "Browser automation CLI for inspecting live pages, prototyping interactions, and running browser workflows."
|
|
4
4
|
license: MIT
|
|
5
5
|
metadata:
|
|
6
6
|
author: saffron-health
|
|
7
|
-
version: "0.
|
|
7
|
+
version: "0.4.0"
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
-
#
|
|
10
|
+
# Libretto
|
|
11
11
|
|
|
12
|
-
Use
|
|
12
|
+
Use `npx libretto` to inspect live browser state, prototype interactions, and run existing browser workflows.
|
|
13
13
|
|
|
14
|
-
##
|
|
14
|
+
## Intro
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
- Use this skill when the truth is on the page.
|
|
17
|
+
- Prefer Libretto when you need to see what the browser is doing, not when you only need to edit source files.
|
|
18
|
+
- Treat Libretto as a session-based workflow: open a page, inspect it, try a focused action, then turn what you learned into code outside the CLI.
|
|
19
|
+
- When building a new integration, prefer reverse-engineering network requests first. Fall back to browser automation when the request path is unclear, too fragile, or blocked by anti-bot systems.
|
|
17
20
|
|
|
18
|
-
|
|
21
|
+
## Setup
|
|
19
22
|
|
|
20
|
-
-
|
|
21
|
-
- Use `
|
|
22
|
-
-
|
|
23
|
+
- Ask the user to set up snapshot analysis before relying on `snapshot` for page understanding.
|
|
24
|
+
- Use `npx libretto init` for first-time setup.
|
|
25
|
+
- If they already have credentials, `npx libretto ai configure openai|anthropic|gemini|vertex` is enough.
|
|
23
26
|
|
|
24
|
-
##
|
|
27
|
+
## Rules
|
|
25
28
|
|
|
26
|
-
|
|
29
|
+
- Announce which session you are using and what page you are on.
|
|
30
|
+
- Ask instead of guessing when it is unclear what to click, type, or submit.
|
|
31
|
+
- Use `snapshot` to understand unknown page state before trying multiple selectors.
|
|
32
|
+
- Get explicit user confirmation before mutating actions or replaying network requests that may have side effects.
|
|
33
|
+
- Never run multiple `exec` commands at the same time.
|
|
34
|
+
- Keep the browser session open until the user says the session is done.
|
|
27
35
|
|
|
28
36
|
## Commands
|
|
29
37
|
|
|
30
|
-
|
|
31
|
-
npx libretto open <url> [--headed|--headless] # Launch browser and navigate (headed by default)
|
|
32
|
-
npx libretto exec <code> [--visualize] # Execute Playwright TypeScript code (--visualize enables ghost cursor + highlight)
|
|
33
|
-
npx libretto run <integrationFile> <integrationExport> [--params <json> | --params-file <path>] [--auth-profile <domain>] [--headed|--headless] # Execute integration actions
|
|
34
|
-
npx libretto resume # Resume a paused workflow for the current session
|
|
35
|
-
npx libretto snapshot --objective "<what to find>" [--context "<situational info>"]
|
|
36
|
-
npx libretto save <url|domain> # Save session (cookies, localStorage) to .libretto/profiles/
|
|
37
|
-
npx libretto network # Show last 20 captured network requests
|
|
38
|
-
npx libretto actions # Show last 20 captured user/agent actions
|
|
39
|
-
npx libretto pages # List open pages in the session
|
|
40
|
-
npx libretto close # Close the browser
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
All commands accept `--session <name>` for isolated browser instances (default: `default`).
|
|
44
|
-
Built-in sessions: `default`, `dev-server`, `browser-agent`.
|
|
45
|
-
|
|
46
|
-
## Visualize Mode (`--visualize`)
|
|
47
|
-
|
|
48
|
-
Add `--visualize` to any `exec` command to show a ghost cursor and element highlight before each action executes. Use it when the user wants to see what will be clicked/filled before it happens.
|
|
49
|
-
|
|
50
|
-
## Workflow Pause/Resume (`pause()`)
|
|
51
|
-
|
|
52
|
-
Workflows pause by calling `await pause()` (imported from `"libretto"`). In production (`NODE_ENV=production`) it is a no-op.
|
|
53
|
-
|
|
54
|
-
- There are no pause options to pass at call sites. Pause is session-scoped and resolved from the active session.
|
|
55
|
-
- `npx libretto run ...` waits until the workflow either completes or hits the next `pause()`.
|
|
56
|
-
- On pause, the workflow process stays alive and keeps browser/session state.
|
|
57
|
-
- `npx libretto resume --session <name>` sends resume signal and then waits until completion or the next pause.
|
|
58
|
-
- For multi-pause workflows, call `resume` repeatedly until the workflow completes.
|
|
59
|
-
|
|
60
|
-
## Workflow Failures and Reruns
|
|
61
|
-
|
|
62
|
-
- `npx libretto run` always uses the same failure-inspection behavior; no separate debug flag is needed.
|
|
63
|
-
- On workflow failure, Libretto prints the workflow error and keeps the browser open for inspection.
|
|
64
|
-
- After a failed run, use `npx libretto exec --session <name> "<code>"` to inspect or prototype fixes.
|
|
65
|
-
- Re-running `npx libretto run ... --session <name>` re-runs the workflow for that session.
|
|
66
|
-
- If the same session still has a failed workflow worker, Libretto releases that failed worker process before rerunning.
|
|
67
|
-
|
|
68
|
-
## Globals Available in `exec`
|
|
69
|
-
|
|
70
|
-
`page`, `context`, `state`, `browser`, `networkLog({ last?, filter?, method? })`, `actionLog({ last?, filter?, action?, source? })`, `console`, `fetch`, `Buffer`, `URL`, `setTimeout`, `setInterval`, `clearTimeout`, `clearInterval`
|
|
71
|
-
|
|
72
|
-
The `state` object is scoped to a single `exec` invocation and resets on the next call.
|
|
73
|
-
|
|
74
|
-
## CRITICAL: No try/catch in exec
|
|
75
|
-
|
|
76
|
-
**Never use try/catch or .catch() in exec code.** Let errors throw so they surface as exec failures. When an exec fails, you get the full error message (e.g., "intercepts pointer events", "Timeout 30000ms exceeded") — use that to diagnose the problem and write a corrected exec.
|
|
77
|
-
|
|
78
|
-
**Why:** A try/catch inside exec hides failures from you. A click that times out takes 30 seconds — if you retry it in a loop with try/catch, you'll silently burn minutes on the same broken selector with no way to recover. Without try/catch, the error comes back immediately and you can reason about what went wrong.
|
|
38
|
+
### `open`
|
|
79
39
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
```typescript
|
|
83
|
-
// BAD — silently retries for minutes
|
|
84
|
-
try {
|
|
85
|
-
await btn.click();
|
|
86
|
-
} catch {
|
|
87
|
-
/* retry or ignore */
|
|
88
|
-
}
|
|
89
|
-
|
|
90
|
-
// GOOD — check first, fail fast
|
|
91
|
-
if (await btn.isVisible()) await btn.click();
|
|
92
|
-
|
|
93
|
-
// GOOD — check existence before acting
|
|
94
|
-
if ((await page.locator(".cookie-banner").count()) > 0) {
|
|
95
|
-
await page.locator(".cookie-banner button").click();
|
|
96
|
-
}
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
If an action fails despite an element being visible, you should not keep retrying it. Instead you can try the following debugging steps:
|
|
100
|
-
|
|
101
|
-
1. Take a snapshot to inspect what's covering the element
|
|
102
|
-
2. Try `{ force: true }` to bypass actionability checks
|
|
103
|
-
3. Try a completely different approach (e.g., opening a dialog via a different button)
|
|
104
|
-
|
|
105
|
-
## Workflow: Browse and Interact
|
|
40
|
+
- Open a page before using `exec` or `snapshot`.
|
|
41
|
+
- Use headed mode when the user needs to log in or watch the workflow.
|
|
106
42
|
|
|
107
43
|
```bash
|
|
108
|
-
|
|
109
|
-
npx libretto open https://example.com
|
|
110
|
-
|
|
111
|
-
# Interact with elements
|
|
112
|
-
npx libretto exec "await page.locator('button:has-text(\"Sign in\")').click()"
|
|
113
|
-
npx libretto exec "await page.fill('input[name=\"email\"]', 'user@example.com')"
|
|
114
|
-
|
|
115
|
-
# Understand the page — always provide objective and context
|
|
116
|
-
npx libretto snapshot \
|
|
117
|
-
--objective "Find the sign-in form fields and submit button" \
|
|
118
|
-
--context "Navigated to example.com login page. Expecting email/password inputs and a submit button."
|
|
119
|
-
|
|
120
|
-
# Include relevant network calls in context when debugging API interactions
|
|
121
|
-
npx libretto snapshot \
|
|
122
|
-
--objective "Find why the referral list is empty" \
|
|
123
|
-
--context "Logged into eClinicalWorks. Clicked Open Referrals tab. Table appears but shows no rows. Recent POST to /servlet/AjaxServlet returned 200 but with empty body."
|
|
124
|
-
|
|
125
|
-
# Done
|
|
126
|
-
npx libretto close
|
|
44
|
+
npx libretto open https://example.com --headed
|
|
45
|
+
npx libretto open https://example.com --headless --session debug-example
|
|
127
46
|
```
|
|
128
47
|
|
|
129
|
-
|
|
48
|
+
### `exec`
|
|
130
49
|
|
|
131
|
-
|
|
50
|
+
- Use `exec` for focused inspection and short-lived interaction experiments.
|
|
51
|
+
- Let failures throw. Do not hide `exec` failures with `try/catch`.
|
|
132
52
|
|
|
133
53
|
```bash
|
|
134
|
-
|
|
135
|
-
npx libretto
|
|
136
|
-
|
|
137
|
-
# ... manually log in in the browser window ...
|
|
138
|
-
|
|
139
|
-
# Save the session
|
|
140
|
-
npx libretto save portal.example.com
|
|
141
|
-
|
|
142
|
-
# Next time you open this domain, you'll be logged in automatically
|
|
143
|
-
npx libretto open https://portal.example.com
|
|
54
|
+
npx libretto exec "return await page.url()"
|
|
55
|
+
npx libretto exec "return await page.locator('button').count()"
|
|
56
|
+
npx libretto exec --visualize "await page.locator('button:has-text(\"Continue\")').click()"
|
|
144
57
|
```
|
|
145
58
|
|
|
146
|
-
|
|
59
|
+
### `snapshot`
|
|
147
60
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
3. Wait ~60 seconds for the browser to hit the breakpoint
|
|
153
|
-
4. Use `npx libretto exec` (with `--session browser-agent`) to inspect and prototype fixes
|
|
154
|
-
5. Once the fix works, codify it in source files
|
|
155
|
-
6. Restart the job to verify end-to-end
|
|
156
|
-
|
|
157
|
-
```bash
|
|
158
|
-
# Start job in background
|
|
159
|
-
npx browser-agent start \
|
|
160
|
-
--job-type pull-open-referrals \
|
|
161
|
-
--tenant-slug hhb \
|
|
162
|
-
--params '{"vendorName":"eClinicalWorks"}'
|
|
163
|
-
|
|
164
|
-
# Inspect page state
|
|
165
|
-
npx libretto exec --session browser-agent "return await page.url();"
|
|
166
|
-
npx libretto snapshot --session browser-agent \
|
|
167
|
-
--objective "Find dropdown menus and their current selections" \
|
|
168
|
-
--context "Browser agent hit breakpoint during pull-open-referrals job. Need to inspect dropdown state."
|
|
169
|
-
|
|
170
|
-
# List dropdown options
|
|
171
|
-
npx libretto exec --session browser-agent "return await page.locator('option').allTextContents();"
|
|
172
|
-
|
|
173
|
-
# Test a fix
|
|
174
|
-
npx libretto exec --session browser-agent "await page.locator('.dropdown-trigger').click(); return 'clicked';"
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
## Snapshot — The Primary Observation Tool
|
|
178
|
-
|
|
179
|
-
The `snapshot` command captures a PNG screenshot + HTML and (when `--objective` is provided) runs analysis through the configured AI runtime (`codex`, `claude`, or `gemini` via `npx libretto ai configure ...`). `--context` is optional (but recommended for better results). This is the single way to understand what's on the page — use it any time you need to inspect page structure, find elements, or debug what's happening.
|
|
180
|
-
|
|
181
|
-
**Never use `page.screenshot()` via `exec` to understand the page.** Use the `snapshot` command instead — it captures the screenshot, HTML, and runs analysis with selectors. Raw screenshots give you an image with no analysis; `snapshot` gives you the answer.
|
|
182
|
-
|
|
183
|
-
### What to Put in `--objective`
|
|
184
|
-
|
|
185
|
-
The objective tells the vision agent what you're looking for. Be specific:
|
|
186
|
-
|
|
187
|
-
- "Find the referral status column in the table"
|
|
188
|
-
- "Find the error message or alert preventing form submission"
|
|
189
|
-
- "Identify all dropdown menus on the page and their current selections"
|
|
190
|
-
|
|
191
|
-
### What to Put in `--context`
|
|
192
|
-
|
|
193
|
-
Context gives the vision agent situational awareness. Include:
|
|
194
|
-
|
|
195
|
-
1. **Where you are** — page, step, state (e.g., "On the eClinicalWorks referral list page")
|
|
196
|
-
2. **What you did** — actions taken (e.g., "Clicked 'Open Referrals' tab, selected department 'Cardiology'")
|
|
197
|
-
3. **What you expect** — desired state (e.g., "Expecting a table of open referrals with patient names")
|
|
198
|
-
4. **Relevant selectors** — any CSS selectors, data-testids, or element identifiers you already know about
|
|
199
|
-
5. **Task context** — what the automation is trying to accomplish overall
|
|
200
|
-
6. **Network calls** — any relevant HTTP requests/responses (e.g., "POST /api/referrals returned 200 with empty array")
|
|
61
|
+
- Use `snapshot` as the primary page observation tool.
|
|
62
|
+
- When you want analysis, provide both `--objective` and `--context`.
|
|
63
|
+
- If you only need the PNG and HTML files, omit `--objective`. That runs capture-only mode and skips AI analysis.
|
|
64
|
+
- When using `--objective`, expect analysis to take time. Use a timeout of at least 2 minutes for shell-wrapped calls.
|
|
201
65
|
|
|
202
66
|
```bash
|
|
67
|
+
npx libretto snapshot
|
|
203
68
|
npx libretto snapshot \
|
|
204
|
-
--objective "Find the
|
|
205
|
-
--context "
|
|
206
|
-
|
|
207
|
-
# Debugging example
|
|
69
|
+
--objective "Find the sign-in form and submit button" \
|
|
70
|
+
--context "I just opened the login page and need the email field, password field, and submit button."
|
|
208
71
|
npx libretto snapshot \
|
|
209
|
-
--objective "
|
|
210
|
-
--context "
|
|
211
|
-
```
|
|
212
|
-
|
|
213
|
-
## Inspecting Raw DOM with `exec`
|
|
214
|
-
|
|
215
|
-
When the snapshot doesn't give you enough detail — why an element is hidden, what directives or event handlers it has, how it's styled — use `exec` with `page.evaluate` to query the raw DOM directly.
|
|
216
|
-
|
|
217
|
-
- **`outerHTML`** — See the complete markup of an element including all attributes.
|
|
218
|
-
```bash
|
|
219
|
-
npx libretto exec "const el = await page.locator('#myElement').elementHandle(); return await page.evaluate(el => el.outerHTML.substring(0, 500), el);"
|
|
220
|
-
```
|
|
221
|
-
- **Computed styles / parent chain** — Debug why Playwright can't click an element.
|
|
222
|
-
```bash
|
|
223
|
-
npx libretto exec "const el = await page.locator('#myElement').elementHandle(); return await page.evaluate(el => { const chain = []; let n = el; for (let i = 0; i < 8 && n; i++) { const s = getComputedStyle(n); chain.push({ tag: n.tagName, id: n.id, display: s.display, visibility: s.visibility }); n = n.parentElement; } return chain; }, el);"
|
|
224
|
-
```
|
|
225
|
-
- **Any DOM property** — `page.evaluate` gives you full access: `getBoundingClientRect()`, `dataset`, `children`, `classList`, attached event listeners, etc.
|
|
226
|
-
|
|
227
|
-
## Tips
|
|
228
|
-
|
|
229
|
-
- **Never use `page.screenshot()` via `exec`.** Use `npx libretto snapshot` instead — it captures the viewport plus HTML and returns analyzed output with actionable selectors. The `fullPage` option is especially dangerous — it scrolls the entire page to stitch a screenshot, which can crash JavaScript-heavy pages (especially EMR portals like eClinicalWorks).
|
|
230
|
-
- **Never run `exec` commands in parallel.** Always wait for one `exec` to finish before starting the next. Do not use `run_in_background` for `exec` calls. Running simultaneous `exec` calls opens multiple CDP connections to the same page, which corrupts the page state and kills the browser.
|
|
231
|
-
- `open` requires an available session. If the session is already active, Libretto fails fast and asks you to close the existing session or use a different `--session`.
|
|
232
|
-
- `run` also requires an available session, except for the specific case of a prior failed `run` in the same session; in that case Libretto releases the failed worker and allows rerun.
|
|
233
|
-
- Use `return <value>` in `exec` to print results. Strings print raw; objects print as JSON.
|
|
234
|
-
- For iframe content, access via `page.locator('iframe[name="..."]').contentFrame()`.
|
|
235
|
-
- Multiple sessions allow parallel browser instances: `--session test1`, `--session test2`.
|
|
236
|
-
|
|
237
|
-
## Network Logging
|
|
238
|
-
|
|
239
|
-
Network requests are captured automatically for Libretto-managed browser sessions (for example from `npx libretto open` and `npx libretto run`). Non-static HTTP responses are logged to `.libretto/sessions/<session>/network.jsonl`.
|
|
240
|
-
|
|
241
|
-
### CLI: `npx libretto network`
|
|
242
|
-
|
|
243
|
-
```bash
|
|
244
|
-
npx libretto network # show last 20 requests
|
|
245
|
-
npx libretto network --last 50 # show last 50
|
|
246
|
-
npx libretto network --filter 'referral|patient' # regex filter on URL
|
|
247
|
-
npx libretto network --method POST # filter by HTTP method
|
|
248
|
-
npx libretto network --clear # truncate the log file
|
|
249
|
-
```
|
|
250
|
-
|
|
251
|
-
### In exec: `networkLog()`
|
|
252
|
-
|
|
253
|
-
```bash
|
|
254
|
-
npx libretto exec "return await networkLog()"
|
|
255
|
-
npx libretto exec "return await networkLog({ filter: 'servlet', last: 5 })"
|
|
256
|
-
npx libretto exec "return await networkLog({ method: 'POST' })"
|
|
72
|
+
--objective "Explain why the table is empty" \
|
|
73
|
+
--context "I opened the referrals page and expected rows after applying filters."
|
|
257
74
|
```
|
|
258
75
|
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
**Note:** Network logging works for Libretto-managed sessions. It does not capture requests for external sessions like `--session browser-agent`.
|
|
262
|
-
|
|
263
|
-
## Action Logging
|
|
264
|
-
|
|
265
|
-
Browser actions are captured automatically for Libretto-managed browser sessions (for example from `npx libretto open` and `npx libretto run`). Both user interactions (manual clicks, typing in the headed browser window) and agent actions (programmatic Playwright API calls via `exec`) are logged to `.libretto/sessions/<session>/actions.jsonl` with a `source` field of `'user'` or `'agent'` to distinguish the two.
|
|
266
|
-
|
|
267
|
-
### CLI: `npx libretto actions`
|
|
268
|
-
|
|
269
|
-
```bash
|
|
270
|
-
npx libretto actions # show last 20 actions
|
|
271
|
-
npx libretto actions --last 50 # show last 50
|
|
272
|
-
npx libretto actions --filter 'button|input' # regex filter on selector/value
|
|
273
|
-
npx libretto actions --action click # filter by action type
|
|
274
|
-
npx libretto actions --source user # only manual user actions
|
|
275
|
-
npx libretto actions --source agent # only programmatic agent actions
|
|
276
|
-
npx libretto actions --clear # truncate the log file
|
|
277
|
-
```
|
|
76
|
+
### `run`
|
|
278
77
|
|
|
279
|
-
|
|
78
|
+
- Use `run` to execute an existing Libretto workflow.
|
|
79
|
+
- If the workflow fails, Libretto keeps the browser open. Inspect the failed state with `snapshot` and `exec` before editing code.
|
|
80
|
+
- If the workflow pauses, resume it with `npx libretto resume --session <name>`.
|
|
81
|
+
- Re-run the same workflow after each fix to verify the browser behavior end to end.
|
|
280
82
|
|
|
281
83
|
```bash
|
|
282
|
-
npx libretto
|
|
283
|
-
npx libretto
|
|
284
|
-
npx libretto
|
|
84
|
+
npx libretto run ./integration.ts main
|
|
85
|
+
npx libretto run ./integration.ts main --params '{"status":"open"}'
|
|
86
|
+
npx libretto run ./integration.ts main --auth-profile app.example.com --headed
|
|
285
87
|
```
|
|
286
88
|
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
**Note:** Action logging works for Libretto-managed sessions. It does not capture actions for external sessions like `--session browser-agent`.
|
|
290
|
-
|
|
291
|
-
## Workflow: Creating a New Integration
|
|
292
|
-
|
|
293
|
-
Use Libretto CLI interactively to build a brand new integration from scratch. Navigate the real site with the user, discover the network endpoints, and codify the data extraction into a reusable TypeScript script.
|
|
294
|
-
|
|
295
|
-
**IMPORTANT:** Do NOT explore the codebase or research existing code before starting. This skill file and the CLI commands below contain everything you need. Jump straight into using the CLI interactively — ask the user for the URL, open the browser, and start working. The only exception is if the user mentions a specific file or piece of code to reference — then read that specific file first, but nothing more.
|
|
296
|
-
|
|
297
|
-
### Approach Selection
|
|
298
|
-
|
|
299
|
-
By default, use the **preferred ordering of approaches**: try the network-first approach (`page.evaluate(fetch(...))`) first, then fall back to Playwright DOM automation if that doesn't work (see "Integration Approaches" below).
|
|
300
|
-
|
|
301
|
-
**If the user explicitly specifies an approach**, use it instead.
|
|
302
|
-
|
|
303
|
-
As part of starting the session, silently run a **security posture review** using the probes from `integration-approach-selection.md` (in this skill's directory) to assess the site's bot detection, fetch interception, and security posture. This tells you:
|
|
304
|
-
|
|
305
|
-
- Whether `page.evaluate(fetch(...))` is safe (fetch not patched, no aggressive bot detection)
|
|
306
|
-
- Whether `page.on('response', ...)` interception is viable
|
|
307
|
-
- Whether you need to restrict to DOM-only extraction
|
|
308
|
-
|
|
309
|
-
If the security review reveals that the default network-first approach won't work (e.g., fetch is monkey-patched, aggressive bot detection), **adapt your approach accordingly and tell the user what you found and which approach you're switching to.** You don't need to ask permission to switch — just explain what you discovered and proceed.
|
|
310
|
-
|
|
311
|
-
The user may also share context during the session that changes the approach (e.g., they know the site blocks direct fetch). Adapt as needed.
|
|
312
|
-
|
|
313
|
-
### Handling Approach Mismatches
|
|
314
|
-
|
|
315
|
-
The security review tells you what's _safe_, but not necessarily what _works_ for every endpoint or data source on the site. As you build the integration, you may find that the recommended approach doesn't produce usable data for a specific part of the workflow. When this happens, **explain what you found, adapt your approach** for that specific part, and keep going.
|
|
316
|
-
|
|
317
|
-
Common mismatches:
|
|
318
|
-
|
|
319
|
-
- **Unparseable response format** — The fetch call succeeds but returns a proprietary format (RSC wire protocol, protobuf, encrypted payloads) instead of parseable JSON/XML/HTML.
|
|
320
|
-
- **Data not in API responses** — The data is server-rendered into HTML or computed client-side; no network response contains it.
|
|
321
|
-
- **Endpoint requires unpredictable parameters** — CSRF tokens, request signatures, or session values that rotate and aren't easily extractable.
|
|
322
|
-
|
|
323
|
-
These can surface at any point — the first endpoint you try or the fifteenth. Different parts of the same integration often need different approaches.
|
|
324
|
-
|
|
325
|
-
### Starting the Session
|
|
326
|
-
|
|
327
|
-
The browser stays open indefinitely until explicitly closed with `npx libretto close` or by the user closing the window. **Do not** set any timeouts, auto-close timers, or call `close` until the user says the workflow session is done. Ensure that you open the browser in `--headed` mode so the user can see what's happening.
|
|
328
|
-
|
|
329
|
-
If the site requires login, ask the user how auth should work in the generated workflow:
|
|
330
|
-
|
|
331
|
-
1. Save a local profile (recommended for local runs): open in `--headed`, have the user log in manually, run `npx libretto save <domain>`, and pass `--auth-profile <domain>` when running the workflow (e.g. `npx libretto run ./file.ts main --auth-profile example.com`).
|
|
332
|
-
2. Use user-managed credential logic in Playwright code (no local profile dependency).
|
|
333
|
-
|
|
334
|
-
If local profile is chosen, include this warning in your generated workflow guidance: local profiles are machine-local (other users/environments will not have them), and sessions can expire so re-login/re-save may be required.
|
|
335
|
-
|
|
336
|
-
### Integration Approaches
|
|
337
|
-
|
|
338
|
-
There are two main approaches for building an integration. **Try the network-first approach first** — it's faster, more reliable, and less brittle. Fall back to Playwright automation if it doesn't work. Be flexible — different parts of the same integration may use different approaches, and a single workflow often mixes them. The user can also explicitly tell you which approach to use.
|
|
339
|
-
|
|
340
|
-
#### Approach 1: Network-First — `page.evaluate(() => fetch(...))` (Try First)
|
|
89
|
+
## Examples
|
|
341
90
|
|
|
342
|
-
|
|
91
|
+
### Building new browser automation workflows
|
|
343
92
|
|
|
344
|
-
|
|
93
|
+
#### Interactive building
|
|
345
94
|
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
If direct fetch calls don't work, fall back to driving the UI with Playwright — clicking elements, filling forms, reading text from the DOM.
|
|
359
|
-
|
|
360
|
-
**How to try it:**
|
|
361
|
-
|
|
362
|
-
1. Navigate to the page.
|
|
363
|
-
2. Use `npx libretto snapshot` to find selectors.
|
|
364
|
-
3. Drive the UI with Playwright locators (`page.locator(...).click()`, `.fill()`, `.textContent()`, etc.).
|
|
365
|
-
|
|
366
|
-
This works regardless of the site's architecture but is slower and more fragile against layout changes.
|
|
367
|
-
|
|
368
|
-
**Supplementing with `page.on('response', ...)`:** When using Playwright automation, you can optionally listen to network responses the browser makes as you navigate — `page.on('response', ...)` lets you capture API data that flows through the site's own code without making extra requests. This is useful when the site has API endpoints but blocks direct fetch calls. Set up listeners before the navigation that triggers the requests. Not all sites will have useful responses to intercept — some are entirely server-rendered.
|
|
369
|
-
|
|
370
|
-
**The workflow for form submissions and data-heavy actions:**
|
|
371
|
-
|
|
372
|
-
1. Use Playwright to fill out the form, select dropdowns, check boxes — whatever the UI requires
|
|
373
|
-
2. **Ask the user for confirmation before submitting** — describe what you're about to submit and wait for approval
|
|
374
|
-
3. Submit the form — network requests are captured automatically (see "Network Logging" above)
|
|
375
|
-
4. Check the captured requests with `npx libretto network --method POST` or `networkLog()`
|
|
376
|
-
5. Inspect the captured request (URL, method, headers, body) to understand the payload structure
|
|
377
|
-
6. Test recreating that request directly via `page.evaluate(() => fetch(...))` — confirm with the user before sending
|
|
378
|
-
7. In the generated production code, skip the form-filling steps and fire the network request directly, parameterized with the relevant input values
|
|
379
|
-
|
|
380
|
-
### Discovering Network Endpoints
|
|
381
|
-
|
|
382
|
-
Network requests are captured automatically in the background (see "Network Logging" above). Use the network log to discover endpoints instead of manually attaching listeners.
|
|
383
|
-
|
|
384
|
-
```bash
|
|
385
|
-
# Fill out a form
|
|
386
|
-
npx libretto exec "await page.locator('#department').selectOption('Cardiology'); return 'selected';"
|
|
387
|
-
npx libretto exec "await page.locator('#status').selectOption('Open'); return 'selected';"
|
|
388
|
-
|
|
389
|
-
# ASK THE USER before submitting — describe what will be submitted
|
|
390
|
-
# Then submit and check what requests fired
|
|
391
|
-
npx libretto exec "await page.locator('#submitBtn').click(); await page.waitForTimeout(3000); return 'submitted';"
|
|
392
|
-
npx libretto network --method POST --last 5
|
|
393
|
-
|
|
394
|
-
# Or query the log programmatically
|
|
395
|
-
npx libretto exec "return await networkLog({ method: 'POST', last: 5 })"
|
|
95
|
+
```text
|
|
96
|
+
<example>
|
|
97
|
+
[Context: The user wants to build a new browser workflow and does not yet know the page structure]
|
|
98
|
+
Assistant: Let me open the target page in headed mode so we can inspect the real workflow.
|
|
99
|
+
Assistant: [Runs `npx libretto open https://target.example.com --headed`]
|
|
100
|
+
Assistant: I'll first determine whether this flow can be built from captured network requests instead of UI-only automation.
|
|
101
|
+
Assistant: [Runs `npx libretto snapshot --objective "Find the next required action" --context "We are starting the workflow from the landing page."`]
|
|
102
|
+
Assistant: If the page exposes a clear and stable request path, I'll prioritize that. If the request path is unclear, fragile, or heavily defended, I'll fall back to browser automation.
|
|
103
|
+
Assistant: [Uses `network`, `snapshot`, and `exec` as needed to prove the approach]
|
|
104
|
+
Assistant: I found the working path. I'll now update the workflow file outside Libretto and verify it with `npx libretto run ...`.
|
|
105
|
+
</example>
|
|
396
106
|
```
|
|
397
107
|
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
```
|
|
401
|
-
|
|
402
|
-
|
|
108
|
+
#### Network requests
|
|
109
|
+
|
|
110
|
+
```text
|
|
111
|
+
<example>
|
|
112
|
+
[Context: The user wants to build an integration using network requests]
|
|
113
|
+
Assistant: [Reads `references/reverse-engineering-network-requests.md`]
|
|
114
|
+
Assistant: Let me open the page in headed mode. Perform the workflow and I'll use the network log to recreate it.
|
|
115
|
+
Assistant: [Runs `npx libretto open https://target.example.com --headed`]
|
|
116
|
+
[User performs workflow]
|
|
117
|
+
User: I've completed the workflow
|
|
118
|
+
Assistant: [Runs `npx libretto network --method POST --last 20`]
|
|
119
|
+
Assistant: I found the relevant requests. I'll recreate the workflow from those requests, then test the resulting script with `npx libretto run ...`.
|
|
120
|
+
</example>
|
|
403
121
|
```
|
|
404
122
|
|
|
405
|
-
###
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
npx libretto
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
418
|
-
body: 'action=getReferrals&department=Cardiology&status=Open'
|
|
419
|
-
});
|
|
420
|
-
return await r.text();
|
|
421
|
-
});
|
|
422
|
-
return resp.substring(0, 1000);
|
|
423
|
-
"
|
|
424
|
-
|
|
425
|
-
# Extract session variables (safe — reads window properties, no server call)
|
|
426
|
-
npx libretto exec "
|
|
427
|
-
return await page.evaluate(() => ({
|
|
428
|
-
sessionDID: (window as any).sessionDID,
|
|
429
|
-
userId: (window as any).TrUserId
|
|
430
|
-
}));
|
|
431
|
-
"
|
|
123
|
+
### Debugging existing workflows
|
|
124
|
+
|
|
125
|
+
```text
|
|
126
|
+
<example>
|
|
127
|
+
[Context: The user has an existing Libretto workflow that is failing]
|
|
128
|
+
Assistant: I'll reproduce the failure first so we can inspect the exact browser state it leaves behind.
|
|
129
|
+
Assistant: [Runs `npx libretto run ./integration.ts main --session debug-flow --headed`]
|
|
130
|
+
Assistant: The workflow failed and Libretto kept the browser open. I'll inspect the page state before changing code.
|
|
131
|
+
Assistant: [Runs `npx libretto snapshot --session debug-flow --objective "Find the blocking error or broken selector target" --context "The workflow just failed after trying to continue from the review step, and I need to identify the visible blocker on the current page."`]
|
|
132
|
+
Assistant: [Runs `npx libretto exec --session debug-flow "...focused inspection or prototype..."`]
|
|
133
|
+
Assistant: I found the issue. I'll patch the workflow code, then rerun `npx libretto run ...` to verify the fix.
|
|
134
|
+
</example>
|
|
432
135
|
```
|
|
433
136
|
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
After completing interactive exploration, **always generate the TypeScript workflow file before ending the session** — do not wait for the user to ask.
|
|
437
|
-
|
|
438
|
-
**STOP AND ASK BEFORE GENERATING CODE.** Once the interactive workflow is figured out, pause and ask:
|
|
439
|
-
|
|
440
|
-
1. "Are there any existing files or patterns in the codebase you want me to reference?"
|
|
441
|
-
2. Check the action log for user interactions by running `npx libretto actions --source user`. If there are any recorded user interactions, ask: "I see you performed some manual interactions in the browser (clicks, form fills, etc.). Would you like me to incorporate any of those into the generated code?" — and briefly list what you found. If there are no user interactions, skip this question entirely.
|
|
442
|
-
3. "Any other guidance for how the production code should be structured?"
|
|
443
|
-
|
|
444
|
-
Wait for the user's response before proceeding. Then:
|
|
445
|
-
|
|
446
|
-
1. **Read `code-generation-rules.md`** (in this skill's directory) — this is mandatory before writing any code. It contains the authoritative rules for Playwright locator usage, `page.evaluate()` restrictions, network request patterns, and type checking. Do not generate code from memory; always reference this file first.
|
|
447
|
-
2. Run the TypeScript type checker against the file and fix any errors before presenting it as done.
|
|
448
|
-
|
|
449
|
-
## Patient Safety Warning
|
|
450
|
-
|
|
451
|
-
Browser automation jobs process real patient health information. The `npx libretto` CLI executes arbitrary code with full page access. **Never execute mutating actions without explicit user confirmation first** (submits, sends, deletes, updates, or other side effects).
|
|
137
|
+
## References
|
|
452
138
|
|
|
453
|
-
For
|
|
139
|
+
- For reverse-engineering captured requests, read `references/reverse-engineering-network-requests.md`.
|
|
140
|
+
- For incorporating manual browser steps the user performed, read `references/user-action-log.md`.
|
|
141
|
+
- For saving and reusing login state, read `references/auth-profiles.md`.
|
|
142
|
+
- For multiple open pages and page targeting, read `references/pages-and-page-targeting.md`.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Auth Profiles
|
|
2
|
+
|
|
3
|
+
Use this reference when the target site requires login and the user wants to reuse local authenticated browser state.
|
|
4
|
+
|
|
5
|
+
## When to Use This
|
|
6
|
+
|
|
7
|
+
- The site requires manual login.
|
|
8
|
+
- The user is running workflows locally.
|
|
9
|
+
- Reusing a saved session is simpler than building credential-handling logic into the workflow.
|
|
10
|
+
|
|
11
|
+
## Workflow
|
|
12
|
+
|
|
13
|
+
- Open the site in headed mode.
|
|
14
|
+
- Ask the user to log in manually.
|
|
15
|
+
- Save the current session as a profile.
|
|
16
|
+
- Reopen the site or run the workflow with that profile.
|
|
17
|
+
|
|
18
|
+
## Commands
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npx libretto open https://app.example.com --headed
|
|
22
|
+
npx libretto save app.example.com
|
|
23
|
+
npx libretto run ./integration.ts main --auth-profile app.example.com
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Notes
|
|
27
|
+
|
|
28
|
+
- Profiles are local to the current machine.
|
|
29
|
+
- Sessions can expire. If the profile stops working, repeat the login and save flow.
|
|
30
|
+
- Keep auth profiles as a brief operational detail in the main skill, not a full workflow pattern.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Pages and Page Targeting
|
|
2
|
+
|
|
3
|
+
Use this reference when a Libretto session has multiple open pages and you need to inspect or target the right one.
|
|
4
|
+
|
|
5
|
+
## When to Use This
|
|
6
|
+
|
|
7
|
+
- The workflow opens a popup, new tab, or secondary page.
|
|
8
|
+
- `exec` or `snapshot` fails because more than one page is open.
|
|
9
|
+
- You are not sure which page in the session holds the relevant state.
|
|
10
|
+
|
|
11
|
+
## Workflow
|
|
12
|
+
|
|
13
|
+
- List the open pages in the session.
|
|
14
|
+
- Identify the page you want by URL.
|
|
15
|
+
- Re-run the command against that page.
|
|
16
|
+
|
|
17
|
+
## Commands
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
npx libretto pages --session debug-flow
|
|
21
|
+
npx libretto exec --session debug-flow --page <page-id> "return await page.url()"
|
|
22
|
+
npx libretto snapshot --session debug-flow --page <page-id> --objective "Find the active form"
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Notes
|
|
26
|
+
|
|
27
|
+
- A session can contain more than one page.
|
|
28
|
+
- When multiple pages are open, think about page targeting first before debugging selectors.
|
|
29
|
+
- Use `pages` to resolve the correct page id, then pass `--page` to `exec`, `snapshot`, `network`, or `actions` when needed.
|