@sous-io/sous 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/README.md +154 -0
- package/bin/run.js +17 -0
- package/bin/xcv +5 -0
- package/package.json +81 -0
- package/shared-prompts/_partials/resume-task.md +51 -0
- package/shared-prompts/_partials/sub-agent-delegation.md +32 -0
- package/shared-prompts/_partials/update-task-file.md +52 -0
- package/shared-prompts/memories/automated-browser-tasks/INDEX.tpl.md +52 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/SKILL.tpl.md +102 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/auth-failure-handling.mjs +81 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/chained-workflow.mjs +126 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/simple-fetch.mjs +92 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/architecture.md +61 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/auth-and-sessions.md +65 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/ctx-api.md +96 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/installation.md +104 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/script-conventions.md +243 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/chrome-state.mjs +148 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/debug.mjs +383 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/debug.spec.mjs +267 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/eslint.config.mjs +56 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/harness.mjs +169 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/keyring.mjs +59 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/logger.mjs +25 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/params.mjs +140 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/run.mjs +140 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/settings.tpl.mjs +1 -0
- package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/utils.mjs +185 -0
- package/shared-prompts/skills/automated-browser-tasks/create-automated-browser-task/SKILL.tpl.md +52 -0
- package/shared-prompts/skills/automated-browser-tasks/running-automated-browser-tasks/SKILL.tpl.md +59 -0
- package/shared-prompts/skills/automated-browser-tasks/update-automated-browser-task/SKILL.tpl.md +47 -0
- package/shared-prompts/skills/control-flow/approve/SKILL.tpl.md +26 -0
- package/shared-prompts/skills/control-flow/opine/SKILL.tpl.md +58 -0
- package/shared-prompts/skills/control-flow/repeat/SKILL.tpl.md +27 -0
- package/shared-prompts/skills/control-flow/research/SKILL.tpl.md +34 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/SKILL.tpl.md +177 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/examples/about-something.md +45 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/examples/do-something.md +33 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/references/advanced-patterns.md +87 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/references/commands.md +46 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/references/frontmatter.md +25 -0
- package/shared-prompts/skills/sous-skills/about-agent-skills/references/substitutions.md +50 -0
- package/shared-prompts/skills/sous-skills/about-liquid-templates/SKILL.tpl.md +268 -0
- package/shared-prompts/skills/sous-skills/about-liquid-templates/references/liquid-filters.md +82 -0
- package/shared-prompts/skills/sous-skills/about-sous/SKILL.tpl.md +51 -0
- package/shared-prompts/skills/sous-skills/create-skill/SKILL.tpl.md +114 -0
- package/shared-prompts/skills/task-files/about-task-files/SKILL.tpl.md +122 -0
- package/shared-prompts/skills/task-files/continue-task-in-new-branch/SKILL.tpl.md +80 -0
- package/shared-prompts/skills/task-files/go/SKILL.tpl.md +14 -0
- package/shared-prompts/skills/task-files/resume-task/SKILL.tpl.md +13 -0
- package/shared-prompts/skills/task-files/start-task/SKILL.tpl.md +93 -0
- package/shared-prompts/skills/task-files/update/SKILL.tpl.md +14 -0
- package/shared-prompts/skills/task-files/update-task-file/SKILL.tpl.md +13 -0
- package/src/base-command.ts +163 -0
- package/src/commands/build.ts +196 -0
- package/src/commands/clear.ts +71 -0
- package/src/commands/compile.ts +95 -0
- package/src/commands/launch.ts +111 -0
- package/src/commands/prune.ts +48 -0
- package/src/lib/build-service.ts +258 -0
- package/src/lib/config-discovery.ts +199 -0
- package/src/lib/env-local.ts +195 -0
- package/src/lib/include-resolver.ts +146 -0
- package/src/lib/markdown-compiler.ts +580 -0
- package/src/lib/pid-service.ts +88 -0
- package/src/lib/settings.ts +695 -0
- package/src/lib/state.ts +135 -0
- package/src/lib/watch-service.ts +115 -0
- package/src/templating/filters/bullet-list.ts +9 -0
- package/src/templating/filters/index.ts +8 -0
- package/src/templating/init-liquid-engine.ts +82 -0
- package/src/templating/lib/glob-files.ts +74 -0
- package/src/templating/lib/import-export.ts +32 -0
- package/src/templating/lib/tag-args.ts +19 -0
- package/src/templating/tags/exportScalarVarsJs.ts +43 -0
- package/src/templating/tags/getFiles.ts +89 -0
- package/src/templating/tags/index.ts +14 -0
- package/src/templating/tags/listFiles.ts +54 -0
- package/src/templating/tags/showVars.ts +22 -0
- package/src/utils/formatting.ts +338 -0
- package/src/utils/prompts.ts +19 -0
package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/SKILL.tpl.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: about-automated-browser-tasks
|
|
3
|
+
description: >
|
|
4
|
+
YOU MUST load this skill when the user asks you to do anything that requires
|
|
5
|
+
interacting with a web browser, or when no MCP server or API can accomplish a
|
|
6
|
+
task and the only path forward is browser-based actions. Before telling the
|
|
7
|
+
user to do something manually in a browser, check for an existing automation
|
|
8
|
+
script and offer to run or write one. When browser work is needed, look for an
|
|
9
|
+
existing script first; create one only if none exists.
|
|
10
|
+
user-invocable: false
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
# Automated Browser Tasks
|
|
14
|
+
|
|
15
|
+
This system lets you write, run, and reuse headless browser automation scripts
|
|
16
|
+
(Playwright) that authenticate as the user by borrowing their existing Chrome
|
|
17
|
+
session — no manual login, no visible browser, Chrome stays open.
|
|
18
|
+
|
|
19
|
+
When a task needs a browser and no API/MCP server can do it, this is the default
|
|
20
|
+
path. Do not instruct the user to click through a browser manually if the task
|
|
21
|
+
can be scripted.
|
|
22
|
+
|
|
23
|
+
## How It Works (cold-start orientation)
|
|
24
|
+
|
|
25
|
+
1. **Auth is automatic.** A harness reads cookies from the user's Chrome profile,
|
|
26
|
+
decrypts them locally (via the OS keyring), and injects them into a fresh
|
|
27
|
+
headless browser context. Scripts "just work" as the logged-in user.
|
|
28
|
+
2. **Scripts are small modules.** Each script file exports `meta` (name,
|
|
29
|
+
description, params) and `execute(ctx)`. The harness resolves + validates
|
|
30
|
+
params, builds `ctx`, and runs `execute`.
|
|
31
|
+
3. **You run scripts by absolute path** through the runner:
|
|
32
|
+
```bash
|
|
33
|
+
node <scriptsDir>/run.mjs <absolute-path-to-script.mjs> [--param=value ...]
|
|
34
|
+
```
|
|
35
|
+
4. **Scripts return a result object**; the runner prints it and writes any
|
|
36
|
+
`outputFile`. Auth failures are detected and reported with an actionable
|
|
37
|
+
message telling the user to log in, then you retry.
|
|
38
|
+
|
|
39
|
+
The framework code (harness, chrome-state, keyring, logger, utils, params,
|
|
40
|
+
run) lives in this skill's `scripts/` directory. User scripts live in the
|
|
41
|
+
project's configured scripts directory and are invoked by path.
|
|
42
|
+
|
|
43
|
+
## The `ctx` Object
|
|
44
|
+
|
|
45
|
+
Every script's `execute(ctx)` receives:
|
|
46
|
+
|
|
47
|
+
- `ctx.page` — Playwright `Page`
|
|
48
|
+
- `ctx.context` / `ctx.browser` — Playwright context / browser
|
|
49
|
+
- `ctx.params` — fully resolved + validated params
|
|
50
|
+
- `ctx.settings` — compiled project settings (from `settings.mjs`)
|
|
51
|
+
- `ctx.timeout` — overall timeout (ms)
|
|
52
|
+
- `ctx.logger` — prefixed logger; `ctx.logger.child('section')` for sections
|
|
53
|
+
- `ctx.utils` — generic action helpers (modal dismissal, waits, text extraction, …)
|
|
54
|
+
- `ctx.debug` — exploration & debugging tools (dump, describe, findText, watch, …)
|
|
55
|
+
- `ctx.checkAuth()` — throws `AuthError` if redirected to a login page
|
|
56
|
+
- `ctx.runChild(script, params)` — run another script in the same browser context
|
|
57
|
+
|
|
58
|
+
Full details: [references/ctx-api.md](references/ctx-api.md).
|
|
59
|
+
|
|
60
|
+
## Writing Scripts — Non-Negotiable Conventions
|
|
61
|
+
|
|
62
|
+
- `execute()` is a thin orchestrator; logic lives in small named step functions.
|
|
63
|
+
- EVERY function (including `execute`) has a full JSDoc block (`@param`/`@returns`).
|
|
64
|
+
- Destructure what you need off `ctx`/`params` at the top of each function.
|
|
65
|
+
- Use `ctx.logger`, never `console.log`.
|
|
66
|
+
- Do NOT validate params in the script — declare rules in `meta.params`.
|
|
67
|
+
- Fail fast; do not catch-and-continue. Fix root causes, not symptoms.
|
|
68
|
+
|
|
69
|
+
Full conventions: [references/script-conventions.md](references/script-conventions.md).
|
|
70
|
+
|
|
71
|
+
## Reference Files
|
|
72
|
+
|
|
73
|
+
- [references/architecture.md](references/architecture.md) — components and data flow
|
|
74
|
+
- [references/ctx-api.md](references/ctx-api.md) — complete `ctx` and `ctx.utils` API
|
|
75
|
+
- [references/auth-and-sessions.md](references/auth-and-sessions.md) — how auth works, auth failures, retries
|
|
76
|
+
- [references/script-conventions.md](references/script-conventions.md) — full scriptwriting rules
|
|
77
|
+
- [references/installation.md](references/installation.md) — dependencies and setup
|
|
78
|
+
|
|
79
|
+
## Examples
|
|
80
|
+
|
|
81
|
+
- [examples/simple-fetch.mjs](examples/simple-fetch.mjs) — single-page data extraction
|
|
82
|
+
- [examples/chained-workflow.mjs](examples/chained-workflow.mjs) — a script calling another via `ctx.runChild`
|
|
83
|
+
- [examples/auth-failure-handling.mjs](examples/auth-failure-handling.mjs) — the auth-failure + retry pattern
|
|
84
|
+
|
|
85
|
+
# Other Skills
|
|
86
|
+
|
|
87
|
+
The agent performing the work MUST load `create-automated-browser-task` when
|
|
88
|
+
creating a new automation script, `update-automated-browser-task` when modifying an
|
|
89
|
+
existing script, and `running-automated-browser-tasks` when running a script.
|
|
90
|
+
|
|
91
|
+
Per the sub-agent delegation pattern
|
|
92
|
+
(`~sous-shared/_partials/sub-agent-delegation.md`), all three are delegated to a background
|
|
93
|
+
sub-agent, which loads the skills itself and reports back. Auth failures and other
|
|
94
|
+
user-facing messages go to the orchestrator to relay; sub-agents cannot talk to the
|
|
95
|
+
user.
|
|
96
|
+
|
|
97
|
+
## Source for this Skill
|
|
98
|
+
|
|
99
|
+
This skill was pulled from the `sous` project's "shared skills" library. It was compiled from a template and
|
|
100
|
+
the output file should not be edited directly.
|
|
101
|
+
|
|
102
|
+
- Source Path: {{ sousTemplatePath }}
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* auth-failure-handling
|
|
3
|
+
*
|
|
4
|
+
* Demonstrates the auth-failure contract. Scripts do NOT log in and do NOT
|
|
5
|
+
* recover mid-run: they call `ctx.checkAuth()` after navigation and let the
|
|
6
|
+
* harness convert a login-page redirect into a structured auth error. The agent
|
|
7
|
+
* relays the message, the user logs in, and the SAME command is re-run.
|
|
8
|
+
*
|
|
9
|
+
* This script targets a page that requires authentication and simply reports
|
|
10
|
+
* whether it reached real content — illustrating where checkAuth belongs.
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
export const meta = {
|
|
14
|
+
name: 'auth-failure-handling',
|
|
15
|
+
description: 'Visit an authenticated page and confirm we are logged in (auth-pattern demo).',
|
|
16
|
+
params: {
|
|
17
|
+
url: {
|
|
18
|
+
required: true,
|
|
19
|
+
description:
|
|
20
|
+
'Absolute URL of a page that requires authentication. If the Chrome ' +
|
|
21
|
+
'session is expired, the harness will surface an actionable auth error.',
|
|
22
|
+
validate: /^https?:\/\/.+/,
|
|
23
|
+
invalidMessage: 'url must be an absolute http(s) URL.',
|
|
24
|
+
},
|
|
25
|
+
readySelector: {
|
|
26
|
+
required: true,
|
|
27
|
+
description:
|
|
28
|
+
'CSS selector for an element that only appears once the authenticated ' +
|
|
29
|
+
'content has loaded (e.g. a user avatar or app shell). Used to confirm we ' +
|
|
30
|
+
'are past any login wall.',
|
|
31
|
+
},
|
|
32
|
+
},
|
|
33
|
+
};
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* Entry point. Navigates, checks auth, then confirms authed content rendered.
|
|
37
|
+
*
|
|
38
|
+
* @param {object} ctx - The harness context.
|
|
39
|
+
* @returns {Promise<object>} `{ found, url, message }`.
|
|
40
|
+
*/
|
|
41
|
+
export async function execute(ctx) {
|
|
42
|
+
const { page, params } = ctx;
|
|
43
|
+
const { url, readySelector } = params;
|
|
44
|
+
|
|
45
|
+
await openAuthedPage(ctx, url, readySelector);
|
|
46
|
+
|
|
47
|
+
return {
|
|
48
|
+
found: true,
|
|
49
|
+
url: page.url(),
|
|
50
|
+
message: 'Reached authenticated content successfully.',
|
|
51
|
+
};
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
/**
|
|
55
|
+
* Navigate to an authenticated page and confirm we reached it — the canonical
|
|
56
|
+
* auth-handling pattern. Many SPAs resolve auth client-side a beat AFTER load and
|
|
57
|
+
* redirect to a login page, so we do NOT check the URL immediately. Instead we
|
|
58
|
+
* wait for the success signal (`readySelector`, an element that only renders for
|
|
59
|
+
* authenticated users). Only if that wait times out do we call `checkAuth()`: by
|
|
60
|
+
* then the URL has settled, so a login page becomes a clear AuthError (which the
|
|
61
|
+
* harness reports as `{ error: 'auth', message }`), while anything else re-throws
|
|
62
|
+
* as a genuine render timeout. The script never tries to log in itself.
|
|
63
|
+
*
|
|
64
|
+
* @param {object} ctx - The harness context.
|
|
65
|
+
* @param {string} url - The authenticated URL to load.
|
|
66
|
+
* @param {string} readySelector - Element proving authed content rendered.
|
|
67
|
+
* @returns {Promise<void>}
|
|
68
|
+
*/
|
|
69
|
+
async function openAuthedPage(ctx, url, readySelector) {
|
|
70
|
+
const { page, logger, timeout, checkAuth } = ctx;
|
|
71
|
+
const log = logger.child('navigate');
|
|
72
|
+
log.info(`Loading authenticated page ${url}`);
|
|
73
|
+
await page.goto(url, { waitUntil: 'domcontentloaded', timeout });
|
|
74
|
+
try {
|
|
75
|
+
await page.locator(readySelector).first().waitFor({ state: 'visible', timeout });
|
|
76
|
+
log.info('Authenticated content is present');
|
|
77
|
+
} catch (renderTimeout) {
|
|
78
|
+
await checkAuth(); // throws AuthError if the settled URL is a login page
|
|
79
|
+
throw renderTimeout;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* chained-workflow
|
|
3
|
+
*
|
|
4
|
+
* Demonstrates composing scripts: one script invokes another via
|
|
5
|
+
* `ctx.runChild`, reusing the same authenticated browser context. The child
|
|
6
|
+
* runs in-process and returns its result object directly.
|
|
7
|
+
*
|
|
8
|
+
* Here the parent collects a list of item URLs from an index page, then runs a
|
|
9
|
+
* child "fetch" script against each one. In a real project the child would be a
|
|
10
|
+
* separate script file imported at the top; it is inlined here so the example
|
|
11
|
+
* is self-contained.
|
|
12
|
+
*/
|
|
13
|
+
|
|
14
|
+
export const meta = {
|
|
15
|
+
name: 'chained-workflow',
|
|
16
|
+
description: 'Collect links from an index page, then fetch each via a child script.',
|
|
17
|
+
params: {
|
|
18
|
+
indexUrl: {
|
|
19
|
+
required: true,
|
|
20
|
+
description:
|
|
21
|
+
'Absolute URL of the index/listing page to scrape links from. Loaded as ' +
|
|
22
|
+
'the authenticated user.',
|
|
23
|
+
validate: /^https?:\/\/.+/,
|
|
24
|
+
invalidMessage: 'indexUrl must be an absolute http(s) URL.',
|
|
25
|
+
},
|
|
26
|
+
linkSelector: {
|
|
27
|
+
required: false,
|
|
28
|
+
default: 'a[href]',
|
|
29
|
+
description:
|
|
30
|
+
'CSS selector matching the links to follow on the index page. Defaults ' +
|
|
31
|
+
'to all anchors with an href.',
|
|
32
|
+
},
|
|
33
|
+
limit: {
|
|
34
|
+
required: false,
|
|
35
|
+
default: 3,
|
|
36
|
+
description:
|
|
37
|
+
'Maximum number of links to follow, to keep runs bounded. Provided as a ' +
|
|
38
|
+
'string on the CLI and coerced to a number.',
|
|
39
|
+
validate: (v) => Number(v) > 0 || 'limit must be a positive number',
|
|
40
|
+
},
|
|
41
|
+
},
|
|
42
|
+
};
|
|
43
|
+
|
|
44
|
+
/** A small child script run once per collected link. Normally its own file. */
|
|
45
|
+
const fetchTitle = {
|
|
46
|
+
meta: {
|
|
47
|
+
name: 'fetch-title',
|
|
48
|
+
params: {
|
|
49
|
+
url: { required: true, validate: /^https?:\/\/.+/, invalidMessage: 'url must be absolute http(s)' },
|
|
50
|
+
},
|
|
51
|
+
},
|
|
52
|
+
/**
|
|
53
|
+
* Load a URL and return its document title.
|
|
54
|
+
*
|
|
55
|
+
* @param {object} ctx - The harness context (shares the parent's browser).
|
|
56
|
+
* @returns {Promise<object>} `{ url, title }`.
|
|
57
|
+
*/
|
|
58
|
+
async execute(ctx) {
|
|
59
|
+
const { page, params, logger, timeout, checkAuth } = ctx;
|
|
60
|
+
logger.child('navigate').info(`Fetching ${params.url}`);
|
|
61
|
+
// domcontentloaded is enough here: the <title> is in the initial document.
|
|
62
|
+
await page.goto(params.url, { waitUntil: 'domcontentloaded', timeout });
|
|
63
|
+
await checkAuth();
|
|
64
|
+
return { url: page.url(), title: await page.title() };
|
|
65
|
+
},
|
|
66
|
+
};
|
|
67
|
+
|
|
68
|
+
/**
|
|
69
|
+
* Entry point. Collects links from the index, then fetches each via a child.
|
|
70
|
+
*
|
|
71
|
+
* @param {object} ctx - The harness context.
|
|
72
|
+
* @returns {Promise<object>} `{ found, count, results }`.
|
|
73
|
+
*/
|
|
74
|
+
export async function execute(ctx) {
|
|
75
|
+
const { params } = ctx;
|
|
76
|
+
const { indexUrl, linkSelector, limit } = params;
|
|
77
|
+
|
|
78
|
+
const links = await collectLinks(ctx, indexUrl, linkSelector, Number(limit));
|
|
79
|
+
const results = await fetchEach(ctx, links);
|
|
80
|
+
|
|
81
|
+
return { found: results.length > 0, count: results.length, results };
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/**
|
|
85
|
+
* Load the index page and collect up to `limit` link URLs.
|
|
86
|
+
*
|
|
87
|
+
* @param {object} ctx - The harness context.
|
|
88
|
+
* @param {string} indexUrl - The listing page URL.
|
|
89
|
+
* @param {string} selector - Selector matching links to follow.
|
|
90
|
+
* @param {number} limit - Max links to return.
|
|
91
|
+
* @returns {Promise<string[]>} Absolute link URLs.
|
|
92
|
+
*/
|
|
93
|
+
async function collectLinks(ctx, indexUrl, selector, limit) {
|
|
94
|
+
const { page, logger, timeout, checkAuth } = ctx;
|
|
95
|
+
const log = logger.child('collect');
|
|
96
|
+
log.info(`Loading index ${indexUrl}`);
|
|
97
|
+
await page.goto(indexUrl, { waitUntil: 'domcontentloaded', timeout });
|
|
98
|
+
// Success signal: the links we intend to read are present. If they never
|
|
99
|
+
// appear, check auth (login redirects resolve late) before failing.
|
|
100
|
+
try {
|
|
101
|
+
await page.locator(selector).first().waitFor({ state: 'visible', timeout });
|
|
102
|
+
} catch (renderTimeout) {
|
|
103
|
+
await checkAuth();
|
|
104
|
+
throw renderTimeout;
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
const hrefs = await page.$$eval(selector, (els) => els.map((el) => el.href));
|
|
108
|
+
const unique = [...new Set(hrefs.filter(Boolean))].slice(0, limit);
|
|
109
|
+
log.info(`Collected ${unique.length} link(s)`);
|
|
110
|
+
return unique;
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
/**
|
|
114
|
+
* Run the child fetch script against each link, in sequence.
|
|
115
|
+
*
|
|
116
|
+
* @param {object} ctx - The harness context.
|
|
117
|
+
* @param {string[]} links - URLs to fetch.
|
|
118
|
+
* @returns {Promise<object[]>} One child result per link.
|
|
119
|
+
*/
|
|
120
|
+
async function fetchEach(ctx, links) {
|
|
121
|
+
const results = [];
|
|
122
|
+
for (const url of links) {
|
|
123
|
+
results.push(await ctx.runChild(fetchTitle, { url }));
|
|
124
|
+
}
|
|
125
|
+
return results;
|
|
126
|
+
}
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* simple-fetch
|
|
3
|
+
*
|
|
4
|
+
* The minimal shape of an automation script: navigate to one page, confirm we
|
|
5
|
+
* are authenticated, and extract a piece of text. Use this as the starting
|
|
6
|
+
* template for single-page data extraction.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
export const meta = {
|
|
10
|
+
name: 'simple-fetch',
|
|
11
|
+
description: 'Navigate to a single page and extract its <h1> (demonstration script).',
|
|
12
|
+
params: {
|
|
13
|
+
url: {
|
|
14
|
+
required: true,
|
|
15
|
+
description:
|
|
16
|
+
'The absolute URL to load. Must include the scheme (https://). The page ' +
|
|
17
|
+
'is loaded as the authenticated user via injected Chrome cookies.',
|
|
18
|
+
validate: /^https?:\/\/.+/,
|
|
19
|
+
invalidMessage: 'url must be an absolute http(s) URL, e.g. "https://example.com".',
|
|
20
|
+
},
|
|
21
|
+
selector: {
|
|
22
|
+
required: false,
|
|
23
|
+
default: 'h1',
|
|
24
|
+
description:
|
|
25
|
+
'CSS selector for the element whose text to extract. Defaults to the ' +
|
|
26
|
+
'first <h1>. The longest matching element\'s text is returned.',
|
|
27
|
+
},
|
|
28
|
+
outputFile: {
|
|
29
|
+
required: false,
|
|
30
|
+
description:
|
|
31
|
+
'Optional absolute path. When set, the runner writes the extracted text ' +
|
|
32
|
+
'here instead of printing the result object.',
|
|
33
|
+
validate: /^\//,
|
|
34
|
+
invalidMessage: 'outputFile must be an absolute path (starting with "/").',
|
|
35
|
+
},
|
|
36
|
+
},
|
|
37
|
+
};
|
|
38
|
+
|
|
39
|
+
/**
|
|
40
|
+
* Entry point. Loads the page, verifies auth, and extracts the target text.
|
|
41
|
+
*
|
|
42
|
+
* @param {object} ctx - The harness context.
|
|
43
|
+
* @returns {Promise<object>} `{ found, url, content, outputFile }`.
|
|
44
|
+
*/
|
|
45
|
+
export async function execute(ctx) {
|
|
46
|
+
const { page, params } = ctx;
|
|
47
|
+
const { url, selector, outputFile } = params;
|
|
48
|
+
|
|
49
|
+
await loadPage(ctx, url, selector);
|
|
50
|
+
const content = await extractText(ctx, selector);
|
|
51
|
+
|
|
52
|
+
return { found: content !== null, url: page.url(), content, outputFile };
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Load a URL and wait for the target element to render. Uses `domcontentloaded`
|
|
57
|
+
* (NOT `networkidle`, which is not a readiness signal) and then waits for the
|
|
58
|
+
* specific thing we need — the target selector. If that times out, we check auth:
|
|
59
|
+
* a login redirect (resolved late by many apps) surfaces as a clear AuthError,
|
|
60
|
+
* otherwise the render timeout is re-thrown.
|
|
61
|
+
*
|
|
62
|
+
* @param {object} ctx - The harness context.
|
|
63
|
+
* @param {string} url - Absolute URL to load.
|
|
64
|
+
* @param {string} selector - The element we expect to render (our success signal).
|
|
65
|
+
* @returns {Promise<void>}
|
|
66
|
+
*/
|
|
67
|
+
async function loadPage(ctx, url, selector) {
|
|
68
|
+
const { page, logger, timeout, checkAuth } = ctx;
|
|
69
|
+
const log = logger.child('navigate');
|
|
70
|
+
log.info(`Loading ${url}`);
|
|
71
|
+
await page.goto(url, { waitUntil: 'domcontentloaded', timeout });
|
|
72
|
+
try {
|
|
73
|
+
await page.locator(selector).first().waitFor({ state: 'visible', timeout });
|
|
74
|
+
} catch (renderTimeout) {
|
|
75
|
+
await checkAuth(); // throws AuthError if the settled URL is a login page
|
|
76
|
+
throw renderTimeout;
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* Extract the longest text matching a selector.
|
|
82
|
+
*
|
|
83
|
+
* @param {object} ctx - The harness context.
|
|
84
|
+
* @param {string} selector - CSS selector to read.
|
|
85
|
+
* @returns {Promise<string|null>} The text, or null if nothing matched.
|
|
86
|
+
*/
|
|
87
|
+
async function extractText(ctx, selector) {
|
|
88
|
+
const { utils, logger } = ctx;
|
|
89
|
+
const text = await utils.extractLongestText([selector], { minLength: 1 });
|
|
90
|
+
logger.child('extract').info(text ? `Got ${text.length} chars` : 'No match');
|
|
91
|
+
return text;
|
|
92
|
+
}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
The system has six framework modules (in this skill's `scripts/`) plus the
|
|
4
|
+
user's own scripts (in the project's configured scripts directory).
|
|
5
|
+
|
|
6
|
+
## Components
|
|
7
|
+
|
|
8
|
+
1. **`keyring.mjs`** — Reads Chrome's Safe Storage password from the OS keyring
|
|
9
|
+
(GNOME keyring) via the D-Bus Secret Service API. Pure JS (`dbus-next`), no
|
|
10
|
+
native deps, no Python. Exports `getChromeSafeStoragePassword()`.
|
|
11
|
+
|
|
12
|
+
2. **`chrome-state.mjs`** — Reads cookies from Chrome's `Cookies` SQLite DB
|
|
13
|
+
(read-only; Chrome stays open), decrypts them with a PBKDF2 key derived from
|
|
14
|
+
the keyring password, and builds a Playwright `storageState` object. Exports
|
|
15
|
+
`extractCookies()`, `buildStorageState()`, `listProfiles()`.
|
|
16
|
+
|
|
17
|
+
3. **`logger.mjs`** — Plain prefixed-line logger. `createLogger(prefix)` returns
|
|
18
|
+
`{ info, warn, error, child }`; `child(section)` extends the prefix.
|
|
19
|
+
|
|
20
|
+
4. **`utils.mjs`** — `createUtils(page, logger)` builds the `ctx.utils` toolkit,
|
|
21
|
+
bound to the live page (modal dismissal, waits, text extraction, screenshot,
|
|
22
|
+
retry, …). See `ctx-api.md`.
|
|
23
|
+
|
|
24
|
+
5. **`params.mjs`** — `resolveParams(meta, explicit, settings)` resolves params
|
|
25
|
+
in priority order and validates them, throwing `ParamError` on failure.
|
|
26
|
+
|
|
27
|
+
6. **`harness.mjs`** — `runScript(script, params, options)`. Resolves/validates
|
|
28
|
+
params, extracts cookies, launches headless Chromium with the injected
|
|
29
|
+
storageState, builds `ctx`, runs `execute(ctx)` under an overall timeout, and
|
|
30
|
+
returns a result envelope. Defines `AuthError` and `ctx.checkAuth()`.
|
|
31
|
+
|
|
32
|
+
7. **`run.mjs`** — CLI entry point. Parses `--key=value` args into harness
|
|
33
|
+
options vs. script params, loads compiled `settings.mjs`, imports the target
|
|
34
|
+
script by absolute path, prints resolved params, runs, reports, writes any
|
|
35
|
+
`outputFile`.
|
|
36
|
+
|
|
37
|
+
## Data Flow
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
run.mjs
|
|
41
|
+
├─ parse args → { params, options }
|
|
42
|
+
├─ load settings.mjs → ctx.settings
|
|
43
|
+
├─ import <script>.mjs
|
|
44
|
+
└─ runScript(script, params, { ...options, settings })
|
|
45
|
+
├─ resolveParams(meta, params, settings) ← validation gate
|
|
46
|
+
├─ chrome-state.buildStorageState()
|
|
47
|
+
│ └─ keyring.getChromeSafeStoragePassword()
|
|
48
|
+
├─ chromium.launch({ headless }) + newContext({ storageState })
|
|
49
|
+
├─ build ctx { page, params, settings, logger, utils, checkAuth, runChild }
|
|
50
|
+
└─ script.execute(ctx) → result
|
|
51
|
+
└─ report result, write outputFile
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Result Envelope
|
|
55
|
+
|
|
56
|
+
`runScript` always returns `{ success, ... }`:
|
|
57
|
+
|
|
58
|
+
- success → `{ success: true, data: <script return value> }`
|
|
59
|
+
- param failure → `{ success: false, error: 'params', message }`
|
|
60
|
+
- auth failure → `{ success: false, error: 'auth', message, url, indicators }`
|
|
61
|
+
- other failure → `{ success: false, error: 'script', message, stack }`
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Auth & Sessions
|
|
2
|
+
|
|
3
|
+
Scripts run as the logged-in user without any manual login step, and without
|
|
4
|
+
closing or controlling the user's running Chrome.
|
|
5
|
+
|
|
6
|
+
## How auth works
|
|
7
|
+
|
|
8
|
+
1. The harness reads the user's Chrome `Cookies` SQLite DB **read-only** — Chrome
|
|
9
|
+
can stay open; there is no lock conflict.
|
|
10
|
+
2. Cookie values are encrypted. The harness fetches Chrome's "Safe Storage"
|
|
11
|
+
password from the OS keyring (GNOME keyring) over the D-Bus Secret Service
|
|
12
|
+
API, derives an AES key (PBKDF2, salt `saltysalt`, 1 iteration, SHA1, 16
|
|
13
|
+
bytes), and decrypts each cookie.
|
|
14
|
+
- **v10**: AES-128-CBC, IV = 16 spaces.
|
|
15
|
+
- **v11**: AES-128-CBC, IV embedded in bytes 3–18; strip a 16-byte random
|
|
16
|
+
prefix from the decrypted plaintext.
|
|
17
|
+
3. Decrypted cookies become a Playwright `storageState`, injected into a fresh
|
|
18
|
+
headless context. The browser is now authenticated as the user.
|
|
19
|
+
|
|
20
|
+
This is all local. Nothing leaves the machine. No Python; pure JS via
|
|
21
|
+
`dbus-next`, `better-sqlite3`, and Node's `crypto`.
|
|
22
|
+
|
|
23
|
+
## Choosing the profile
|
|
24
|
+
|
|
25
|
+
The Chrome profile defaults to `Default`. Override per run with
|
|
26
|
+
`--profileName="Profile 1"`, or set a project default in settings
|
|
27
|
+
(`chromeProfile`) so it flows in via `ctx.settings`.
|
|
28
|
+
|
|
29
|
+
## Detecting auth failures
|
|
30
|
+
|
|
31
|
+
Cookies expire; SSO sessions lapse. After each navigation a script should call
|
|
32
|
+
`await ctx.checkAuth()`. It inspects the current URL for login indicators
|
|
33
|
+
(`/login`, `/sso`, `multipass`, `accounts.google.com`, etc.) and throws an
|
|
34
|
+
`AuthError` if found.
|
|
35
|
+
|
|
36
|
+
The harness catches `AuthError` and returns:
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
{ success: false, error: 'auth', message, url, indicators }
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The `message` is actionable and meant to reach the user verbatim. A sub-agent
|
|
43
|
+
returns it to the orchestrator, which relays it:
|
|
44
|
+
|
|
45
|
+
> Authentication required... Log in to the target site in your Chrome browser
|
|
46
|
+
> (profile: "Default"), then retry this script.
|
|
47
|
+
|
|
48
|
+
## The retry contract
|
|
49
|
+
|
|
50
|
+
There is no mid-script recovery. On an auth failure:
|
|
51
|
+
|
|
52
|
+
1. The message reaches the user, who logs into the site in their Chrome profile.
|
|
53
|
+
2. Re-run the **same** command. The harness re-reads the now-valid cookies.
|
|
54
|
+
|
|
55
|
+
A sub-agent cannot wait for a login, so it stops at step 1 and reports; the
|
|
56
|
+
orchestrator relays the message and dispatches the re-run.
|
|
57
|
+
|
|
58
|
+
Never attempt to script the login itself, and never weaken `checkAuth` to get
|
|
59
|
+
past a login wall.
|
|
60
|
+
|
|
61
|
+
## Scope cookie extraction
|
|
62
|
+
|
|
63
|
+
Extraction can be limited to domain substrings for speed/privacy. The harness
|
|
64
|
+
`domains` option (and `extractCookies(profile, domains)`) filters
|
|
65
|
+
`host_key LIKE '%domain%'`. Leave null to extract all cookies.
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# The `ctx` API
|
|
2
|
+
|
|
3
|
+
Every script's `execute(ctx)` receives a single context object. Destructure what
|
|
4
|
+
you need at the top of each function.
|
|
5
|
+
|
|
6
|
+
## Core members
|
|
7
|
+
|
|
8
|
+
| Member | Type | Description |
|
|
9
|
+
|--------|------|-------------|
|
|
10
|
+
| `ctx.page` | `Page` | Playwright page. Primary interaction surface. |
|
|
11
|
+
| `ctx.context` | `BrowserContext` | The browser context (cookies already injected). |
|
|
12
|
+
| `ctx.browser` | `Browser` | The Chromium instance. |
|
|
13
|
+
| `ctx.params` | object | Fully resolved + validated params. |
|
|
14
|
+
| `ctx.settings` | object | Compiled project settings (from `settings.mjs`). |
|
|
15
|
+
| `ctx.timeout` | number | Overall timeout (ms); use for `goto`/wait calls. |
|
|
16
|
+
| `ctx.logger` | Logger | Prefixed logger (see below). |
|
|
17
|
+
| `ctx.utils` | object | Generic action helpers (see below). |
|
|
18
|
+
| `ctx.debug` | object | Exploration & debugging tools (see below). |
|
|
19
|
+
| `ctx.checkAuth()` | fn | Throws `AuthError` if the current URL looks like a login page. Call after every navigation. |
|
|
20
|
+
| `ctx.runChild(script, params)` | fn | Run another script module in the same browser context; returns its result. Child gets its own resolved params, logger, and utils. |
|
|
21
|
+
|
|
22
|
+
## `ctx.logger`
|
|
23
|
+
|
|
24
|
+
Plain prefixed lines. `info` → stdout; `warn`/`error` → stderr, tagged.
|
|
25
|
+
|
|
26
|
+
```js
|
|
27
|
+
const log = ctx.logger.child('navigate'); // prefix: [script-name:navigate]
|
|
28
|
+
log.info('Loading repo'); // [script-name:navigate] Loading repo
|
|
29
|
+
log.warn('modal not found, using Escape'); // [script-name:navigate] WARN ...
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## `ctx.utils`
|
|
33
|
+
|
|
34
|
+
Built per-run, bound to the live `page` and the script's logger. All are async
|
|
35
|
+
unless noted.
|
|
36
|
+
|
|
37
|
+
| Helper | Signature | Description |
|
|
38
|
+
|--------|-----------|-------------|
|
|
39
|
+
| `autoDismiss` | `(triggerLocator, dismiss)` | **Preferred** modal handling: register a `page.addLocatorHandler` so `dismiss` runs whenever `triggerLocator` blocks an action. Timing-independent — no sleeps, no races. Use for any modal/banner that may appear asynchronously. |
|
|
40
|
+
| `dismissModals` | `(selectors, { timeout? })` | One-shot best-effort dismissal of modals KNOWN to be present already. Clicks the first of each candidate selector that currently exists; no fixed delays. For async modals, use `autoDismiss`. |
|
|
41
|
+
| `clickByText` | `(text, { force?, timeout? })` | Wait for and click the first element matching `text` (RegExp or string). `force` defaults **false** (auto-waits for actionability). |
|
|
42
|
+
| `waitForText` | `(text, { timeout? })` | Wait for text to appear; does not click. |
|
|
43
|
+
| `extractLongestText` | `(selectors, { minLength? })` | Longest `textContent` across candidate selectors meeting `minLength`, or null. |
|
|
44
|
+
| `extractTextByPattern` | `(pattern, { source?, group? })` | First regex match from page body (or supplied `source`); returns trimmed match or null. |
|
|
45
|
+
| `screenshot` | `(path, { fullPage? })` | Save a debug screenshot. Failures are logged, never thrown. |
|
|
46
|
+
| `scrollIntoView` | `(selector)` | Scroll matching element into view; returns boolean found. |
|
|
47
|
+
| `retry` | `(fn, { attempts?, delay?, label? })` | Retry an async fn; throws the last error if exhausted. |
|
|
48
|
+
|
|
49
|
+
### On `force` clicks
|
|
50
|
+
|
|
51
|
+
Non-forced clicks are the resilient default: Playwright auto-waits for the element
|
|
52
|
+
to be visible, stable, and not covered before clicking — which naturally waits out
|
|
53
|
+
overlays and transitions. `force: true` bypasses those checks and is fire-and-forget
|
|
54
|
+
(it can "succeed" while landing on nothing), so reserve it for the rare element a
|
|
55
|
+
component library (e.g. Blueprint.js) wrongly reports as disabled — and verify the
|
|
56
|
+
outcome with a follow-up wait. There is no `waitForSPA` helper: do not wait on
|
|
57
|
+
`networkidle` as a readiness signal (see `script-conventions.md` → Waits); wait for
|
|
58
|
+
the specific element/text you need instead.
|
|
59
|
+
|
|
60
|
+
## `ctx.debug`
|
|
61
|
+
|
|
62
|
+
Tools for **observing the real page** so you build waits and selectors from facts,
|
|
63
|
+
not guesses. Conventions require observing a page before automating it
|
|
64
|
+
(`script-conventions.md` → "observe the real page"); these are how. Built per-run,
|
|
65
|
+
bound to the live `page`/`logger`. All are async and none throw (a debug aid must
|
|
66
|
+
never break a run). Artifacts are written under `ctx.debug.dir` (a unique per-run
|
|
67
|
+
directory under the OS temp dir, created lazily on first write).
|
|
68
|
+
|
|
69
|
+
| Tool | Signature | Description |
|
|
70
|
+
|------|-----------|-------------|
|
|
71
|
+
| `screenshot` | `(name?, { fullPage? })` | Save a PNG; returns its path (or null). |
|
|
72
|
+
| `html` | `(name?)` | Save the current page HTML; returns its path. |
|
|
73
|
+
| `dump` | `(name?)` | Full snapshot — URL, title, visible text, screenshot, HTML — to the debug dir; returns `{ dir, url, title, files, textPreview }`. The go-to "what does this page look like right now?" tool. |
|
|
74
|
+
| `describe` | `(selector, { limit? })` | What a CSS selector matches: `{ count, elements[] }` with tag/role/id/classes/text/visibility/href. Fastest way to check a selector is right. |
|
|
75
|
+
| `count` | `(selector)` | Number of matches. |
|
|
76
|
+
| `clickables` | `({ limit? })` | List visible interactive elements (anchors, buttons, role buttons/tabs/links, onclick). Use when the obvious `<a>`/button doesn't exist and the real control is a click-handled div. |
|
|
77
|
+
| `findText` | `(pattern, { limit? })` | Where does this text live? For each match, the owning element AND its clickable ancestor — i.e. *what to click* to act on that text. Matches text on an element's own text nodes (works even when the element also has icon/element children). |
|
|
78
|
+
| `watch` | `(fn, { samples?, intervalMs?, label? })` | Sample a browser-side metric over time and log how it evolves. Answers "WHEN is this ready?" — e.g. `watch(() => document.body.innerText.length)` exposes that `networkidle` fired on an empty shell. Returns `[{ t, value }]`. |
|
|
79
|
+
| `dir` | getter | Absolute path to this run's artifact directory. |
|
|
80
|
+
|
|
81
|
+
### Auto-capture on failure
|
|
82
|
+
|
|
83
|
+
When a script throws an unexpected error, the harness automatically runs
|
|
84
|
+
`debug.dump('failure')` before closing the browser and reports the directory in the
|
|
85
|
+
result (`result.debugDir`, printed by the runner). So a failed run leaves a
|
|
86
|
+
screenshot, HTML, and text snapshot of the exact failing state with no extra code.
|
|
87
|
+
|
|
88
|
+
### The intended workflow
|
|
89
|
+
|
|
90
|
+
1. Stuck on a selector or timing? `await ctx.debug.dump()` and look at the files.
|
|
91
|
+
2. Selector returning nothing? `describe(it)` / `clickables()` / `findText(text)`
|
|
92
|
+
to discover the real element (and its clickable ancestor).
|
|
93
|
+
3. Wait firing too early/never? `watch(() => <metric>)` to find the moment the
|
|
94
|
+
page is genuinely ready, then wait on that concrete signal.
|
|
95
|
+
4. Remove debug calls (or leave a couple of cheap ones) once the script is solid;
|
|
96
|
+
`dump`/`screenshot`/`html` write files, so don't leave those in hot loops.
|