@sous-io/sous 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (206) hide show
  1. package/README.md +121 -35
  2. package/bin/run.js +10 -1
  3. package/docs/markdown/README.md +27 -0
  4. package/docs/markdown/_sidebar.md +18 -0
  5. package/docs/markdown/commands.md +308 -0
  6. package/docs/markdown/config-discovery.md +74 -0
  7. package/docs/markdown/config-inspection.md +69 -0
  8. package/docs/markdown/config-layers.md +92 -0
  9. package/docs/markdown/config-variables.md +79 -0
  10. package/docs/markdown/configuration.md +71 -0
  11. package/docs/markdown/design-principles.md +59 -0
  12. package/docs/markdown/repositories-authoring.md +408 -0
  13. package/docs/markdown/repositories-consuming.md +580 -0
  14. package/docs/markdown/repositories-file-formats.md +1084 -0
  15. package/docs/markdown/repositories-variables.md +387 -0
  16. package/docs/markdown/repositories.md +303 -0
  17. package/docs/markdown/skill-categories.md +58 -0
  18. package/package.json +73 -9
  19. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/SKILL.tpl.md +20 -20
  20. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/examples/about-something.md +2 -2
  21. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/examples/do-something.md +1 -1
  22. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/references/advanced-patterns.md +6 -6
  23. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/references/commands.md +5 -5
  24. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/references/frontmatter.md +3 -3
  25. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-liquid-templates/SKILL.tpl.md +40 -25
  26. package/recipes/core/sous-skills/skills/about-sous/SKILL.tpl.md +70 -0
  27. package/recipes/core/sous-skills/skills/about-sous-configuration/SKILL.tpl.md +75 -0
  28. package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/create-skill/SKILL.tpl.md +8 -9
  29. package/recipes/core/sous-skills/sous.recipe.yaml +45 -0
  30. package/sous.config.schema.json +337 -0
  31. package/src/base-command.ts +220 -67
  32. package/src/commands/build.ts +150 -73
  33. package/src/commands/clear.ts +23 -15
  34. package/src/commands/compile.ts +74 -16
  35. package/src/commands/config/get.ts +110 -0
  36. package/src/commands/config/show.ts +32 -0
  37. package/src/commands/config/validate.ts +53 -0
  38. package/src/commands/help.ts +46 -0
  39. package/src/commands/launch.ts +36 -14
  40. package/src/commands/lock/rebuild.ts +241 -0
  41. package/src/commands/lock/show.ts +115 -0
  42. package/src/commands/namespace/list.ts +117 -0
  43. package/src/commands/namespace/show.ts +110 -0
  44. package/src/commands/prune.ts +3 -11
  45. package/src/commands/recipe/list.ts +95 -0
  46. package/src/commands/recipe/show.ts +301 -0
  47. package/src/commands/repo/add.ts +145 -0
  48. package/src/commands/repo/gc.ts +172 -0
  49. package/src/commands/repo/init.ts +136 -0
  50. package/src/commands/repo/link.ts +500 -0
  51. package/src/commands/repo/list.ts +179 -0
  52. package/src/commands/repo/release.ts +619 -0
  53. package/src/commands/repo/remove.ts +193 -0
  54. package/src/commands/repo/search.ts +189 -0
  55. package/src/commands/repo/submit.ts +133 -0
  56. package/src/commands/repo/unlink.ts +147 -0
  57. package/src/commands/subscription/add.ts +285 -0
  58. package/src/commands/subscription/list.ts +129 -0
  59. package/src/commands/subscription/remove.ts +181 -0
  60. package/src/commands/vars/ask.ts +374 -0
  61. package/src/commands/vars/index.ts +79 -0
  62. package/src/commands/vars/list.ts +67 -0
  63. package/src/commands/vars/show.ts +77 -0
  64. package/src/config-command.ts +30 -0
  65. package/src/lib/build-service.ts +206 -54
  66. package/src/lib/config-discovery.ts +220 -27
  67. package/src/lib/config-inspect.ts +145 -0
  68. package/src/lib/config-kernel.mjs +377 -0
  69. package/src/lib/config-schema.ts +361 -0
  70. package/src/lib/env-file.ts +328 -0
  71. package/src/lib/env-local.ts +18 -1
  72. package/src/lib/errors.ts +32 -0
  73. package/src/lib/include-resolver.ts +108 -15
  74. package/src/lib/interactive.ts +165 -0
  75. package/src/lib/markdown-compiler.ts +118 -37
  76. package/src/lib/package-info.ts +25 -0
  77. package/src/lib/pid-service.ts +32 -21
  78. package/src/lib/refs/find.ts +589 -0
  79. package/src/lib/refs/index.ts +12 -0
  80. package/src/lib/refs/pick.ts +147 -0
  81. package/src/lib/refs/scopes.ts +61 -0
  82. package/src/lib/repos/catalog-display.ts +116 -0
  83. package/src/lib/repos/catalog-inputs.ts +160 -0
  84. package/src/lib/repos/catalog.ts +722 -0
  85. package/src/lib/repos/core-recipe.ts +105 -0
  86. package/src/lib/repos/defaults.ts +175 -0
  87. package/src/lib/repos/formats/common.ts +389 -0
  88. package/src/lib/repos/formats/index-file.ts +215 -0
  89. package/src/lib/repos/formats/links-map.ts +96 -0
  90. package/src/lib/repos/formats/lockfile.ts +167 -0
  91. package/src/lib/repos/formats/patterns.ts +57 -0
  92. package/src/lib/repos/formats/recipe-manifest.ts +395 -0
  93. package/src/lib/repos/formats/repo-manifest.ts +88 -0
  94. package/src/lib/repos/formats/store-entry.ts +84 -0
  95. package/src/lib/repos/freshness.ts +208 -0
  96. package/src/lib/repos/git-clone.ts +312 -0
  97. package/src/lib/repos/identity.ts +89 -0
  98. package/src/lib/repos/index.ts +58 -0
  99. package/src/lib/repos/links.ts +353 -0
  100. package/src/lib/repos/load-manifest.ts +236 -0
  101. package/src/lib/repos/lock-service.ts +453 -0
  102. package/src/lib/repos/locked-namespace-resolver.ts +90 -0
  103. package/src/lib/repos/locked-recipes.ts +254 -0
  104. package/src/lib/repos/managed-layer.ts +422 -0
  105. package/src/lib/repos/namespace-resolver.ts +370 -0
  106. package/src/lib/repos/providers/base.ts +206 -0
  107. package/src/lib/repos/providers/git.ts +233 -0
  108. package/src/lib/repos/providers/github.ts +294 -0
  109. package/src/lib/repos/providers/gitlab.ts +263 -0
  110. package/src/lib/repos/providers/http.ts +102 -0
  111. package/src/lib/repos/providers/index-cache.ts +382 -0
  112. package/src/lib/repos/providers/index.ts +106 -0
  113. package/src/lib/repos/providers/local.ts +391 -0
  114. package/src/lib/repos/providers/provider.ts +401 -0
  115. package/src/lib/repos/recipe-config-layers.ts +287 -0
  116. package/src/lib/repos/recipe-targets.ts +223 -0
  117. package/src/lib/repos/ref-search.ts +46 -0
  118. package/src/lib/repos/ref.ts +513 -0
  119. package/src/lib/repos/reference-report.ts +122 -0
  120. package/src/lib/repos/release/bump.ts +161 -0
  121. package/src/lib/repos/release/git-state.ts +305 -0
  122. package/src/lib/repos/release/index-builder.ts +635 -0
  123. package/src/lib/repos/release/index.ts +16 -0
  124. package/src/lib/repos/release/plan.ts +512 -0
  125. package/src/lib/repos/release/submit-service.ts +496 -0
  126. package/src/lib/repos/release/tags.ts +243 -0
  127. package/src/lib/repos/release/validate.ts +463 -0
  128. package/src/lib/repos/resolver.ts +789 -0
  129. package/src/lib/repos/scaffold/index.ts +238 -0
  130. package/src/lib/repos/scaffold/templates.ts +413 -0
  131. package/src/lib/repos/seed.ts +414 -0
  132. package/src/lib/repos/store/contract.ts +64 -0
  133. package/src/lib/repos/store/hash.ts +114 -0
  134. package/src/lib/repos/store/recipe-store.ts +599 -0
  135. package/src/lib/repos/store/settings.ts +58 -0
  136. package/src/lib/repos/subscription-service.ts +2678 -0
  137. package/src/lib/repos/trust.ts +447 -0
  138. package/src/lib/settings.ts +546 -189
  139. package/src/lib/sous-home.ts +104 -0
  140. package/src/lib/state.ts +52 -20
  141. package/src/lib/vars/ask.ts +1152 -0
  142. package/src/lib/vars/definition-source.ts +252 -0
  143. package/src/lib/vars/display.ts +233 -0
  144. package/src/lib/vars/index.ts +18 -0
  145. package/src/lib/vars/ladder.ts +282 -0
  146. package/src/lib/vars/mappings.ts +265 -0
  147. package/src/lib/vars/names.ts +94 -0
  148. package/src/lib/vars/preanswers.ts +395 -0
  149. package/src/lib/vars/question-plan.ts +218 -0
  150. package/src/lib/vars/report.ts +228 -0
  151. package/src/lib/vars/safe-regex.ts +235 -0
  152. package/src/lib/vars/validate.ts +312 -0
  153. package/src/lib/watch-loop.ts +148 -0
  154. package/src/templating/init-liquid-engine.ts +58 -16
  155. package/src/utils/choice-prompt.ts +143 -0
  156. package/src/utils/command-errors.ts +186 -0
  157. package/src/utils/command-help.ts +45 -0
  158. package/src/utils/confirm-prompt.ts +110 -0
  159. package/src/utils/flags.ts +153 -0
  160. package/src/utils/formatting.ts +540 -55
  161. package/src/utils/prompts.ts +35 -1
  162. package/src/utils/sous-directory.ts +245 -0
  163. package/src/utils/table.ts +603 -0
  164. package/src/utils/value-prompt.ts +119 -0
  165. package/bin/xcv +0 -5
  166. package/shared-prompts/_partials/resume-task.md +0 -51
  167. package/shared-prompts/_partials/sub-agent-delegation.md +0 -32
  168. package/shared-prompts/_partials/update-task-file.md +0 -52
  169. package/shared-prompts/memories/automated-browser-tasks/INDEX.tpl.md +0 -52
  170. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/SKILL.tpl.md +0 -102
  171. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/auth-failure-handling.mjs +0 -81
  172. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/chained-workflow.mjs +0 -126
  173. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/examples/simple-fetch.mjs +0 -92
  174. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/architecture.md +0 -61
  175. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/auth-and-sessions.md +0 -65
  176. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/ctx-api.md +0 -96
  177. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/installation.md +0 -104
  178. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/references/script-conventions.md +0 -243
  179. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/chrome-state.mjs +0 -148
  180. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/debug.mjs +0 -383
  181. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/debug.spec.mjs +0 -267
  182. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/eslint.config.mjs +0 -56
  183. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/harness.mjs +0 -169
  184. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/keyring.mjs +0 -59
  185. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/logger.mjs +0 -25
  186. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/params.mjs +0 -140
  187. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/run.mjs +0 -140
  188. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/settings.tpl.mjs +0 -1
  189. package/shared-prompts/skills/automated-browser-tasks/about-automated-browser-tasks/scripts/utils.mjs +0 -185
  190. package/shared-prompts/skills/automated-browser-tasks/create-automated-browser-task/SKILL.tpl.md +0 -52
  191. package/shared-prompts/skills/automated-browser-tasks/running-automated-browser-tasks/SKILL.tpl.md +0 -59
  192. package/shared-prompts/skills/automated-browser-tasks/update-automated-browser-task/SKILL.tpl.md +0 -47
  193. package/shared-prompts/skills/control-flow/approve/SKILL.tpl.md +0 -26
  194. package/shared-prompts/skills/control-flow/opine/SKILL.tpl.md +0 -58
  195. package/shared-prompts/skills/control-flow/repeat/SKILL.tpl.md +0 -27
  196. package/shared-prompts/skills/control-flow/research/SKILL.tpl.md +0 -34
  197. package/shared-prompts/skills/sous-skills/about-sous/SKILL.tpl.md +0 -51
  198. package/shared-prompts/skills/task-files/about-task-files/SKILL.tpl.md +0 -122
  199. package/shared-prompts/skills/task-files/continue-task-in-new-branch/SKILL.tpl.md +0 -80
  200. package/shared-prompts/skills/task-files/go/SKILL.tpl.md +0 -14
  201. package/shared-prompts/skills/task-files/resume-task/SKILL.tpl.md +0 -13
  202. package/shared-prompts/skills/task-files/start-task/SKILL.tpl.md +0 -93
  203. package/shared-prompts/skills/task-files/update/SKILL.tpl.md +0 -14
  204. package/shared-prompts/skills/task-files/update-task-file/SKILL.tpl.md +0 -13
  205. /package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-agent-skills/references/substitutions.md +0 -0
  206. /package/{shared-prompts/skills/sous-skills → recipes/core/sous-skills/skills}/about-liquid-templates/references/liquid-filters.md +0 -0
@@ -1,92 +0,0 @@
1
- /**
2
- * simple-fetch
3
- *
4
- * The minimal shape of an automation script: navigate to one page, confirm we
5
- * are authenticated, and extract a piece of text. Use this as the starting
6
- * template for single-page data extraction.
7
- */
8
-
9
- export const meta = {
10
- name: 'simple-fetch',
11
- description: 'Navigate to a single page and extract its <h1> (demonstration script).',
12
- params: {
13
- url: {
14
- required: true,
15
- description:
16
- 'The absolute URL to load. Must include the scheme (https://). The page ' +
17
- 'is loaded as the authenticated user via injected Chrome cookies.',
18
- validate: /^https?:\/\/.+/,
19
- invalidMessage: 'url must be an absolute http(s) URL, e.g. "https://example.com".',
20
- },
21
- selector: {
22
- required: false,
23
- default: 'h1',
24
- description:
25
- 'CSS selector for the element whose text to extract. Defaults to the ' +
26
- 'first <h1>. The longest matching element\'s text is returned.',
27
- },
28
- outputFile: {
29
- required: false,
30
- description:
31
- 'Optional absolute path. When set, the runner writes the extracted text ' +
32
- 'here instead of printing the result object.',
33
- validate: /^\//,
34
- invalidMessage: 'outputFile must be an absolute path (starting with "/").',
35
- },
36
- },
37
- };
38
-
39
- /**
40
- * Entry point. Loads the page, verifies auth, and extracts the target text.
41
- *
42
- * @param {object} ctx - The harness context.
43
- * @returns {Promise<object>} `{ found, url, content, outputFile }`.
44
- */
45
- export async function execute(ctx) {
46
- const { page, params } = ctx;
47
- const { url, selector, outputFile } = params;
48
-
49
- await loadPage(ctx, url, selector);
50
- const content = await extractText(ctx, selector);
51
-
52
- return { found: content !== null, url: page.url(), content, outputFile };
53
- }
54
-
55
- /**
56
- * Load a URL and wait for the target element to render. Uses `domcontentloaded`
57
- * (NOT `networkidle`, which is not a readiness signal) and then waits for the
58
- * specific thing we need — the target selector. If that times out, we check auth:
59
- * a login redirect (resolved late by many apps) surfaces as a clear AuthError,
60
- * otherwise the render timeout is re-thrown.
61
- *
62
- * @param {object} ctx - The harness context.
63
- * @param {string} url - Absolute URL to load.
64
- * @param {string} selector - The element we expect to render (our success signal).
65
- * @returns {Promise<void>}
66
- */
67
- async function loadPage(ctx, url, selector) {
68
- const { page, logger, timeout, checkAuth } = ctx;
69
- const log = logger.child('navigate');
70
- log.info(`Loading ${url}`);
71
- await page.goto(url, { waitUntil: 'domcontentloaded', timeout });
72
- try {
73
- await page.locator(selector).first().waitFor({ state: 'visible', timeout });
74
- } catch (renderTimeout) {
75
- await checkAuth(); // throws AuthError if the settled URL is a login page
76
- throw renderTimeout;
77
- }
78
- }
79
-
80
- /**
81
- * Extract the longest text matching a selector.
82
- *
83
- * @param {object} ctx - The harness context.
84
- * @param {string} selector - CSS selector to read.
85
- * @returns {Promise<string|null>} The text, or null if nothing matched.
86
- */
87
- async function extractText(ctx, selector) {
88
- const { utils, logger } = ctx;
89
- const text = await utils.extractLongestText([selector], { minLength: 1 });
90
- logger.child('extract').info(text ? `Got ${text.length} chars` : 'No match');
91
- return text;
92
- }
@@ -1,61 +0,0 @@
1
- # Architecture
2
-
3
- The system has six framework modules (in this skill's `scripts/`) plus the
4
- user's own scripts (in the project's configured scripts directory).
5
-
6
- ## Components
7
-
8
- 1. **`keyring.mjs`** — Reads Chrome's Safe Storage password from the OS keyring
9
- (GNOME keyring) via the D-Bus Secret Service API. Pure JS (`dbus-next`), no
10
- native deps, no Python. Exports `getChromeSafeStoragePassword()`.
11
-
12
- 2. **`chrome-state.mjs`** — Reads cookies from Chrome's `Cookies` SQLite DB
13
- (read-only; Chrome stays open), decrypts them with a PBKDF2 key derived from
14
- the keyring password, and builds a Playwright `storageState` object. Exports
15
- `extractCookies()`, `buildStorageState()`, `listProfiles()`.
16
-
17
- 3. **`logger.mjs`** — Plain prefixed-line logger. `createLogger(prefix)` returns
18
- `{ info, warn, error, child }`; `child(section)` extends the prefix.
19
-
20
- 4. **`utils.mjs`** — `createUtils(page, logger)` builds the `ctx.utils` toolkit,
21
- bound to the live page (modal dismissal, waits, text extraction, screenshot,
22
- retry, …). See `ctx-api.md`.
23
-
24
- 5. **`params.mjs`** — `resolveParams(meta, explicit, settings)` resolves params
25
- in priority order and validates them, throwing `ParamError` on failure.
26
-
27
- 6. **`harness.mjs`** — `runScript(script, params, options)`. Resolves/validates
28
- params, extracts cookies, launches headless Chromium with the injected
29
- storageState, builds `ctx`, runs `execute(ctx)` under an overall timeout, and
30
- returns a result envelope. Defines `AuthError` and `ctx.checkAuth()`.
31
-
32
- 7. **`run.mjs`** — CLI entry point. Parses `--key=value` args into harness
33
- options vs. script params, loads compiled `settings.mjs`, imports the target
34
- script by absolute path, prints resolved params, runs, reports, writes any
35
- `outputFile`.
36
-
37
- ## Data Flow
38
-
39
- ```
40
- run.mjs
41
- ├─ parse args → { params, options }
42
- ├─ load settings.mjs → ctx.settings
43
- ├─ import <script>.mjs
44
- └─ runScript(script, params, { ...options, settings })
45
- ├─ resolveParams(meta, params, settings) ← validation gate
46
- ├─ chrome-state.buildStorageState()
47
- │ └─ keyring.getChromeSafeStoragePassword()
48
- ├─ chromium.launch({ headless }) + newContext({ storageState })
49
- ├─ build ctx { page, params, settings, logger, utils, checkAuth, runChild }
50
- └─ script.execute(ctx) → result
51
- └─ report result, write outputFile
52
- ```
53
-
54
- ## Result Envelope
55
-
56
- `runScript` always returns `{ success, ... }`:
57
-
58
- - success → `{ success: true, data: <script return value> }`
59
- - param failure → `{ success: false, error: 'params', message }`
60
- - auth failure → `{ success: false, error: 'auth', message, url, indicators }`
61
- - other failure → `{ success: false, error: 'script', message, stack }`
@@ -1,65 +0,0 @@
1
- # Auth & Sessions
2
-
3
- Scripts run as the logged-in user without any manual login step, and without
4
- closing or controlling the user's running Chrome.
5
-
6
- ## How auth works
7
-
8
- 1. The harness reads the user's Chrome `Cookies` SQLite DB **read-only** — Chrome
9
- can stay open; there is no lock conflict.
10
- 2. Cookie values are encrypted. The harness fetches Chrome's "Safe Storage"
11
- password from the OS keyring (GNOME keyring) over the D-Bus Secret Service
12
- API, derives an AES key (PBKDF2, salt `saltysalt`, 1 iteration, SHA1, 16
13
- bytes), and decrypts each cookie.
14
- - **v10**: AES-128-CBC, IV = 16 spaces.
15
- - **v11**: AES-128-CBC, IV embedded in bytes 3–18; strip a 16-byte random
16
- prefix from the decrypted plaintext.
17
- 3. Decrypted cookies become a Playwright `storageState`, injected into a fresh
18
- headless context. The browser is now authenticated as the user.
19
-
20
- This is all local. Nothing leaves the machine. No Python; pure JS via
21
- `dbus-next`, `better-sqlite3`, and Node's `crypto`.
22
-
23
- ## Choosing the profile
24
-
25
- The Chrome profile defaults to `Default`. Override per run with
26
- `--profileName="Profile 1"`, or set a project default in settings
27
- (`chromeProfile`) so it flows in via `ctx.settings`.
28
-
29
- ## Detecting auth failures
30
-
31
- Cookies expire; SSO sessions lapse. After each navigation a script should call
32
- `await ctx.checkAuth()`. It inspects the current URL for login indicators
33
- (`/login`, `/sso`, `multipass`, `accounts.google.com`, etc.) and throws an
34
- `AuthError` if found.
35
-
36
- The harness catches `AuthError` and returns:
37
-
38
- ```
39
- { success: false, error: 'auth', message, url, indicators }
40
- ```
41
-
42
- The `message` is actionable and meant to reach the user verbatim. A sub-agent
43
- returns it to the orchestrator, which relays it:
44
-
45
- > Authentication required... Log in to the target site in your Chrome browser
46
- > (profile: "Default"), then retry this script.
47
-
48
- ## The retry contract
49
-
50
- There is no mid-script recovery. On an auth failure:
51
-
52
- 1. The message reaches the user, who logs into the site in their Chrome profile.
53
- 2. Re-run the **same** command. The harness re-reads the now-valid cookies.
54
-
55
- A sub-agent cannot wait for a login, so it stops at step 1 and reports; the
56
- orchestrator relays the message and dispatches the re-run.
57
-
58
- Never attempt to script the login itself, and never weaken `checkAuth` to get
59
- past a login wall.
60
-
61
- ## Scope cookie extraction
62
-
63
- Extraction can be limited to domain substrings for speed/privacy. The harness
64
- `domains` option (and `extractCookies(profile, domains)`) filters
65
- `host_key LIKE '%domain%'`. Leave null to extract all cookies.
@@ -1,96 +0,0 @@
1
- # The `ctx` API
2
-
3
- Every script's `execute(ctx)` receives a single context object. Destructure what
4
- you need at the top of each function.
5
-
6
- ## Core members
7
-
8
- | Member | Type | Description |
9
- |--------|------|-------------|
10
- | `ctx.page` | `Page` | Playwright page. Primary interaction surface. |
11
- | `ctx.context` | `BrowserContext` | The browser context (cookies already injected). |
12
- | `ctx.browser` | `Browser` | The Chromium instance. |
13
- | `ctx.params` | object | Fully resolved + validated params. |
14
- | `ctx.settings` | object | Compiled project settings (from `settings.mjs`). |
15
- | `ctx.timeout` | number | Overall timeout (ms); use for `goto`/wait calls. |
16
- | `ctx.logger` | Logger | Prefixed logger (see below). |
17
- | `ctx.utils` | object | Generic action helpers (see below). |
18
- | `ctx.debug` | object | Exploration & debugging tools (see below). |
19
- | `ctx.checkAuth()` | fn | Throws `AuthError` if the current URL looks like a login page. Call after every navigation. |
20
- | `ctx.runChild(script, params)` | fn | Run another script module in the same browser context; returns its result. Child gets its own resolved params, logger, and utils. |
21
-
22
- ## `ctx.logger`
23
-
24
- Plain prefixed lines. `info` → stdout; `warn`/`error` → stderr, tagged.
25
-
26
- ```js
27
- const log = ctx.logger.child('navigate'); // prefix: [script-name:navigate]
28
- log.info('Loading repo'); // [script-name:navigate] Loading repo
29
- log.warn('modal not found, using Escape'); // [script-name:navigate] WARN ...
30
- ```
31
-
32
- ## `ctx.utils`
33
-
34
- Built per-run, bound to the live `page` and the script's logger. All are async
35
- unless noted.
36
-
37
- | Helper | Signature | Description |
38
- |--------|-----------|-------------|
39
- | `autoDismiss` | `(triggerLocator, dismiss)` | **Preferred** modal handling: register a `page.addLocatorHandler` so `dismiss` runs whenever `triggerLocator` blocks an action. Timing-independent — no sleeps, no races. Use for any modal/banner that may appear asynchronously. |
40
- | `dismissModals` | `(selectors, { timeout? })` | One-shot best-effort dismissal of modals KNOWN to be present already. Clicks the first of each candidate selector that currently exists; no fixed delays. For async modals, use `autoDismiss`. |
41
- | `clickByText` | `(text, { force?, timeout? })` | Wait for and click the first element matching `text` (RegExp or string). `force` defaults **false** (auto-waits for actionability). |
42
- | `waitForText` | `(text, { timeout? })` | Wait for text to appear; does not click. |
43
- | `extractLongestText` | `(selectors, { minLength? })` | Longest `textContent` across candidate selectors meeting `minLength`, or null. |
44
- | `extractTextByPattern` | `(pattern, { source?, group? })` | First regex match from page body (or supplied `source`); returns trimmed match or null. |
45
- | `screenshot` | `(path, { fullPage? })` | Save a debug screenshot. Failures are logged, never thrown. |
46
- | `scrollIntoView` | `(selector)` | Scroll matching element into view; returns boolean found. |
47
- | `retry` | `(fn, { attempts?, delay?, label? })` | Retry an async fn; throws the last error if exhausted. |
48
-
49
- ### On `force` clicks
50
-
51
- Non-forced clicks are the resilient default: Playwright auto-waits for the element
52
- to be visible, stable, and not covered before clicking — which naturally waits out
53
- overlays and transitions. `force: true` bypasses those checks and is fire-and-forget
54
- (it can "succeed" while landing on nothing), so reserve it for the rare element a
55
- component library (e.g. Blueprint.js) wrongly reports as disabled — and verify the
56
- outcome with a follow-up wait. There is no `waitForSPA` helper: do not wait on
57
- `networkidle` as a readiness signal (see `script-conventions.md` → Waits); wait for
58
- the specific element/text you need instead.
59
-
60
- ## `ctx.debug`
61
-
62
- Tools for **observing the real page** so you build waits and selectors from facts,
63
- not guesses. Conventions require observing a page before automating it
64
- (`script-conventions.md` → "observe the real page"); these are how. Built per-run,
65
- bound to the live `page`/`logger`. All are async and none throw (a debug aid must
66
- never break a run). Artifacts are written under `ctx.debug.dir` (a unique per-run
67
- directory under the OS temp dir, created lazily on first write).
68
-
69
- | Tool | Signature | Description |
70
- |------|-----------|-------------|
71
- | `screenshot` | `(name?, { fullPage? })` | Save a PNG; returns its path (or null). |
72
- | `html` | `(name?)` | Save the current page HTML; returns its path. |
73
- | `dump` | `(name?)` | Full snapshot — URL, title, visible text, screenshot, HTML — to the debug dir; returns `{ dir, url, title, files, textPreview }`. The go-to "what does this page look like right now?" tool. |
74
- | `describe` | `(selector, { limit? })` | What a CSS selector matches: `{ count, elements[] }` with tag/role/id/classes/text/visibility/href. Fastest way to check a selector is right. |
75
- | `count` | `(selector)` | Number of matches. |
76
- | `clickables` | `({ limit? })` | List visible interactive elements (anchors, buttons, role buttons/tabs/links, onclick). Use when the obvious `<a>`/button doesn't exist and the real control is a click-handled div. |
77
- | `findText` | `(pattern, { limit? })` | Where does this text live? For each match, the owning element AND its clickable ancestor — i.e. *what to click* to act on that text. Matches text on an element's own text nodes (works even when the element also has icon/element children). |
78
- | `watch` | `(fn, { samples?, intervalMs?, label? })` | Sample a browser-side metric over time and log how it evolves. Answers "WHEN is this ready?" — e.g. `watch(() => document.body.innerText.length)` exposes that `networkidle` fired on an empty shell. Returns `[{ t, value }]`. |
79
- | `dir` | getter | Absolute path to this run's artifact directory. |
80
-
81
- ### Auto-capture on failure
82
-
83
- When a script throws an unexpected error, the harness automatically runs
84
- `debug.dump('failure')` before closing the browser and reports the directory in the
85
- result (`result.debugDir`, printed by the runner). So a failed run leaves a
86
- screenshot, HTML, and text snapshot of the exact failing state with no extra code.
87
-
88
- ### The intended workflow
89
-
90
- 1. Stuck on a selector or timing? `await ctx.debug.dump()` and look at the files.
91
- 2. Selector returning nothing? `describe(it)` / `clickables()` / `findText(text)`
92
- to discover the real element (and its clickable ancestor).
93
- 3. Wait firing too early/never? `watch(() => <metric>)` to find the moment the
94
- page is genuinely ready, then wait on that concrete signal.
95
- 4. Remove debug calls (or leave a couple of cheap ones) once the script is solid;
96
- `dump`/`screenshot`/`html` write files, so don't leave those in hot loops.
@@ -1,104 +0,0 @@
1
- # Installation
2
-
3
- ## Runtime & platform
4
-
5
- - Node.js ≥ 22.
6
- - Linux with a GNOME-keyring-compatible Secret Service (the user's Chrome must
7
- have stored its Safe Storage key there — true after Chrome has run once on a
8
- desktop session with an unlocked keyring).
9
- - Google Chrome installed with at least one profile the user has logged into.
10
-
11
- macOS (Keychain) and Windows (DPAPI) are not yet supported by `keyring.mjs`.
12
-
13
- ## Dependencies
14
-
15
- The runner and harness import these at runtime; the framework does not bundle
16
- them. Install them **at the consuming project's root** — NOT globally.
17
-
18
- Why not global: the scripts use ESM `import 'playwright'`. ESM resolves bare
19
- imports by walking *up* the directory tree from the importing file looking for a
20
- `node_modules`. The compiled runner lives at
21
- `<projectRoot>/.claude/skills/about-automated-browser-tasks/scripts/run.mjs`, so a
22
- `node_modules` at `<projectRoot>` is found by walking up; a global npm install is
23
- never on that resolution path.
24
-
25
- ```bash
26
- cd <projectRoot> # the repo root that contains .claude/skills/
27
- npm install playwright better-sqlite3 dbus-next
28
- npx playwright install chromium
29
- ```
30
-
31
- Add a `package.json` at `<projectRoot>` if none exists (`{"type":"module","private":true}`)
32
- and gitignore `node_modules/`.
33
-
34
- - `playwright` — headless browser automation.
35
- - `better-sqlite3` — reads Chrome's `Cookies` SQLite DB.
36
- - `dbus-next` — pure-JS D-Bus client for the keyring (no Python, no native build).
37
-
38
- Tested with: Playwright 1.61, better-sqlite3 12.x, Node 22, Chrome cookie format
39
- v11, Ubuntu 22.04.
40
-
41
- ## Project wiring (via sous)
42
-
43
- A downstream project compiles this bundle into its skills directory and compiles
44
- `settings.tpl.mjs` → `settings.mjs` (sibling of `run.mjs`) so scripts get
45
- `ctx.settings`. Example compilation targets:
46
-
47
- ```js
48
- compilation: {
49
- targets: [
50
- {
51
- // The skills (SKILL.tpl.md, references, examples, scripts) → skills dir
52
- entryGlob: "${sousRootPath}/shared-prompts/skills/automated-browser-tasks/**/*",
53
- outputs: [{ destinationDir: "${projectRoot}/.claude/skills" }],
54
- },
55
- ],
56
- }
57
- ```
58
-
59
- Define project values (`chromeProfile`, base URLs, resource IDs, …) in `_vars`. The
60
- `{% exportScalarVarsJs %}` tag in `settings.tpl.mjs` emits all in-scope scalars
61
- as the runtime settings module — no per-key wiring needed. Be sure to define
62
- `browserAutomationScriptsDir` (the absolute path to the project's task scripts)
63
- in `_vars` — both the runtime and the task manifest below rely on it.
64
-
65
- ## Task manifest in core memory
66
-
67
- So the agent always knows which browser tasks exist (without relying on a skill
68
- trigger firing), render the shared memory partial into the project's memory source
69
- tree, then `@include` it from a core-memory file. It renders a live list of every
70
- task script via `{% getFiles … import="meta" %}`, reading each script's `meta`.
71
-
72
- `@include` does NOT substitute variables, so you cannot `@`-include the shared
73
- `INDEX.tpl.md` by an absolute `${...}` path. Instead, add a compilation target that
74
- renders it into your memory tree (exactly how `runtimeContext` emits
75
- `session-context.md`):
76
-
77
- ```js
78
- // A target that renders the shared manifest into the project's memory source.
79
- const browserTaskManifest = {
80
- entryPoint: "${sousRootPath}/shared-prompts/memories/automated-browser-tasks/INDEX.tpl.md",
81
- outputs: [
82
- { destinationFile: "${memoryRoot}/tools/automated-browser-tasks.md" },
83
- ],
84
- };
85
- ```
86
-
87
- Order this target BEFORE the memories target that composes core memory. Then pull
88
- it into a memory file (e.g. `tools/README.md`) with a plain relative Sous include:
89
- put an `@`-prefixed line containing just the rendered filename
90
- (`automated-browser-tasks.md`) on its own line in that file.
91
-
92
- The manifest auto-rebuilds on every `xcv build`, so newly created tasks appear
93
- automatically. It requires `browserAutomationScriptsDir` to be in scope (the
94
- absolute path to the task scripts).
95
-
96
- ## Verifying
97
-
98
- Run any example script by absolute path:
99
-
100
- ```bash
101
- node <scriptsDir>/run.mjs <scriptsDir>/../examples/simple-fetch.mjs --url=https://example.com
102
- ```
103
-
104
- A clean run prints extracted cookie counts, a browser-ready line, and the result.
@@ -1,243 +0,0 @@
1
- # Script Conventions
2
-
3
- These rules are non-negotiable. A reviewer (or linter) should be able to reject a
4
- script that violates them.
5
-
6
- ## Quality bar: bulletproof or it doesn't ship
7
-
8
- A flaky script is a broken script. "Works most of the time" is failure. Write for
9
- 100% reliability across many consecutive and parallel runs from the first draft —
10
- do not ship something that "usually works" and plan to harden later.
11
-
12
- The single greatest source of flakiness is **guessing about timing instead of
13
- waiting for facts**. Every wait must key off a concrete, observable condition that
14
- *proves* the thing you need is ready. Spend the extra time to find that signal.
15
- Arbitrary delays (`waitForTimeout`) are the enemy — see [Waits](#waits); they are
16
- effectively banned.
17
-
18
- Before writing a single wait or selector, **observe the real page.** Do not assume
19
- DOM structure. Build your waits from what you actually see — selectors invented
20
- from imagination are how you get a script that passes once and fails in CI.
21
-
22
- `ctx.debug` exists for exactly this (full surface in `ctx-api.md`):
23
-
24
- - `ctx.debug.dump()` — snapshot URL, title, text, screenshot, and HTML to disk.
25
- - `ctx.debug.describe(selector)` / `ctx.debug.clickables()` — see whether a
26
- selector matches and what the real interactive elements are (often a
27
- click-handled `div`, not the `<a>`/`<button>` you assumed).
28
- - `ctx.debug.findText(text)` — locate text and the clickable ancestor to target.
29
- - `ctx.debug.watch(() => metric)` — sample a metric over time to find the *moment*
30
- the page is genuinely ready (this is how you discover that `networkidle` fired
31
- on an empty shell), then wait on that concrete signal.
32
-
33
- On an unexpected throw, the harness auto-captures a failure snapshot
34
- (`result.debugDir`) — check it first when a run fails. Remove file-writing debug
35
- calls (`dump`/`screenshot`/`html`) once the script is solid; keep them out of hot
36
- loops.
37
-
38
- ## Structure
39
-
40
- `execute(ctx)` is a thin orchestrator that reads like a table of contents. All
41
- real work lives in small, named step functions defined below `execute` in the
42
- same file.
43
-
44
- - One discrete action per step function (navigate, dismiss, extract, parse…).
45
- - ≤ 30 lines per function; 10 or fewer is ideal.
46
- - Module-level functions, not class methods.
47
- - Generic patterns → `ctx.utils`. Site-specific patterns → step functions (which
48
- a project may later factor into shared libs it imports).
49
-
50
- ## Doc-blocks
51
-
52
- EVERY function — `execute` included — has a proper JSDoc block: a description
53
- line plus `@param` for every argument and `@returns`. Use
54
- `@returns {Promise<void>}` for functions that return nothing. Single-line
55
- `/** … */` comments are NOT sufficient.
56
-
57
- ## Destructuring
58
-
59
- Each function destructures the members it needs off `ctx` (and off `params`) at
60
- the top of its body, so the body never repeats `ctx.`/`params.` prefixes:
61
-
62
- ```js
63
- async function navigateToRepo(ctx, baseUrl, repoId) {
64
- const { page, logger, timeout, checkAuth } = ctx;
65
- ...
66
- }
67
- ```
68
-
69
- ## Params (`meta.params`)
70
-
71
- The framework resolves and validates params before `execute` runs. Scripts never
72
- validate their own params. Resolution priority (low → high):
73
- `ctx.settings` < `meta.params[x].default` < explicit (CLI) params.
74
-
75
- Each param spec:
76
-
77
- | Field | Type | Meaning |
78
- |-------|------|---------|
79
- | `required` | boolean | Error if nothing resolves. |
80
- | `default` | any | Fallback value. |
81
- | `description` | string | Be genuinely descriptive: what it is, where to find it, how it's used, consequence of omitting. Shown in listings and errors. |
82
- | `validate` | `RegExp` \| `Function` | See below. |
83
- | `invalidMessage` | string | Error for a failing RegExp, or a `validate` fn returning `false`. |
84
-
85
- `validate`:
86
- - **RegExp** — resolved value (as string) must match.
87
- - **Function** `(value, resolvedParams) => true | false | string` — `true` valid;
88
- a returned `string` is used as the error; `false` falls back to `invalidMessage`.
89
- The function gets all resolved params, enabling cross-param checks.
90
-
91
- All failures across params are collected into one `ParamError`.
92
-
93
- ## Logging
94
-
95
- Use `ctx.logger`, never `console.log`. Create a child per section:
96
- `const log = ctx.logger.child('navigate')`. Output is
97
- `[script-name:section] message`. Levels: `info`, `warn`, `error`.
98
-
99
- ## Return shape
100
-
101
- Return a plain object. Common keys:
102
-
103
- - `found: boolean` — whether the target content was located.
104
- - `content: string` — extracted content (when found).
105
- - `outputFile: string` — path for the runner to write `content` to.
106
- - a URL key (e.g. `buildUrl`) — where content was found.
107
- - `message: string` — human-readable explanation, especially on failure.
108
-
109
- On `found: false`, include diagnostics (`pageTextPreview`, `message`).
110
-
111
- ## Verify every action
112
-
113
- Do not assume an action took effect — prove it. After every navigation or click
114
- that changes state, wait for a signal that confirms the *intended outcome*:
115
-
116
- - After a navigation: `await page.waitForURL(/expected-path/)`, or wait for an
117
- element that only exists on the destination.
118
- - After a click that should open a view: wait for that view's content, not just
119
- for the click to return.
120
- - After triggering content load: wait for the content to be present AND non-empty
121
- (e.g. a `<pre>` whose text length exceeds a threshold), not merely attached.
122
-
123
- A click with `{ force: true }` is fire-and-forget: it bypasses Playwright's
124
- actionability checks (visible, stable, not covered) and reports success even when
125
- it lands on nothing. Prefer a plain click — Playwright then auto-waits for the
126
- element to be actionable, which naturally waits out overlays and transitions.
127
- Reserve `force` for the rare element a component library wrongly reports as
128
- disabled, and even then verify the outcome afterward.
129
-
130
- ## Error handling
131
-
132
- Throw on unexpected failures; the harness catches and reports. Never
133
- catch-and-continue to paper over a problem. Auth failures come from
134
- `ctx.checkAuth()`; page-interaction failures (missing element, timeout) should
135
- propagate naturally. Fix root causes, not symptoms.
136
-
137
- **Auth resolves late in SPAs.** A single-page app often loads its shell, *then*
138
- decides client-side that the session is invalid and redirects to a login page a
139
- beat later. So:
140
- - Do NOT call `ctx.checkAuth()` immediately after `goto` — the redirect may not
141
- have happened yet (false pass) and the URL may not have settled.
142
- - Do NOT race the success signal against the login URL — a valid session can
143
- *transiently* touch a login-ish URL before bouncing back (false fail).
144
- - DO wait for your success signal (the authenticated view's element). Only if that
145
- times out, *then* call `ctx.checkAuth()` — by then the URL has settled, so a
146
- login page is a real `AuthError` and anything else is a genuine render timeout.
147
-
148
- ## Naming
149
-
150
- - Files: `verb-noun-qualifier.mjs` (e.g. `get-repo-ci-error.mjs`).
151
- - Step functions: `verbNoun` camelCase (`navigateToRepo`, `dismissModals`).
152
- - Log sections: short, lowercase, no spaces (`navigate`, `dismiss`, `extract`).
153
-
154
- ## Waits
155
-
156
- Wait for **specific, verifiable things** — never for time. This is the rule that
157
- makes scripts bulletproof.
158
-
159
- ### `waitForTimeout` is effectively banned
160
-
161
- A fixed sleep is a bet that something will be ready by then. The bet loses
162
- intermittently — that is precisely what flakiness *is*. Exhaust every avenue for a
163
- condition-based wait before even considering a sleep:
164
-
165
- 1. Wait for an element/state that proves readiness (`locator.waitFor`,
166
- `page.waitForURL`, `expect(locator).toBeVisible()`).
167
- 2. Wait for a content predicate via `page.waitForFunction(() => …)` when readiness
168
- is "the data populated", not just "an element exists".
169
- 3. Wait for a network response (`page.waitForResponse`) when the DOM gives no
170
- signal but a known request does.
171
- 4. Install a handler for interrupting UI (`page.addLocatorHandler`, below) instead
172
- of sleeping to "let a modal pass".
173
-
174
- Only if ALL of these are genuinely impossible may you fall back to
175
- `page.waitForTimeout` — and then you must (a) keep it short, (b) write a comment
176
- explaining what DOM-observable signal you searched for and why none exists, and
177
- (c) feel bad about it. Treat each one as a defect to be removed later. A script
178
- should aim for **zero** `waitForTimeout` calls.
179
-
180
- ### `networkidle` is NOT a readiness signal
181
-
182
- `waitForLoadState('networkidle')` means "the network went quiet", which in a
183
- modern SPA happens long before — or long after — the content you want renders.
184
- A large SPA commonly hits network idle while the DOM is still an empty ~600-char
185
- shell, with every real value still to be fetched and rendered client-side. Never
186
- treat `networkidle` as "the page is ready". Wait for the *specific element or
187
- text* you need instead. Use `domcontentloaded` for the initial `goto`, then a
188
- concrete element wait.
189
-
190
- ### Pick a signal that proves the exact thing you need
191
-
192
- - "Tab bar loaded" → wait for a specific named tab to be visible.
193
- - "List rendered" → wait for a row's distinguishing text (e.g. a commit hash
194
- pattern), not a generic container that exists while empty.
195
- - "Log loaded" → wait for the log element AND a length/content predicate, so an
196
- empty placeholder doesn't satisfy the wait.
197
-
198
- ### Virtualized lists/grids → set a tall viewport, don't scroll-accumulate
199
-
200
- A virtualized list or grid renders only the rows within the scroll viewport (a
201
- 31-row table may put only ~17 rows in the DOM). A single DOM sweep then
202
- silently returns a partial set. The cheap, robust fix is to enlarge the viewport
203
- BEFORE navigating, so the grid materializes every row at once:
204
-
205
- ```js
206
- await page.setViewportSize({ width: 1600, height: 20000 }); // then goto()
207
- ```
208
-
209
- This overrides the harness's default 1920×1080 per-page and needs no harness change.
210
- Prefer it over a scroll-accumulate loop: far less code, no timing loop. Then **verify
211
- completeness** — extract the count the UI advertises (e.g. a "Properties 31" header
212
- badge) and assert the extracted row count equals it, so a clipped read fails loud
213
- instead of returning a silent subset. A tall viewport is not universal: a virtualizer
214
- bounded by its own container's fixed CSS height can still clip regardless of window
215
- size — the assertion is what catches that, and scroll-accumulate is the fallback.
216
-
217
- ### Unpredictable interrupting UI → `addLocatorHandler`, not sleeps
218
-
219
- Modals/banners that appear at an unpredictable moment (welcome dialogs, "what's
220
- new", cookie prompts) are a classic flake source: dismiss-then-continue races the
221
- modal's appearance. Register a handler once; Playwright auto-runs it whenever that
222
- element would block an action — fully timing-independent:
223
-
224
- ```js
225
- await page.addLocatorHandler(
226
- page.getByRole('dialog').filter({ has: page.getByRole('button', { name: 'Close' }) }),
227
- async (dialog) => { await dialog.getByRole('button', { name: 'Close' }).click(); }
228
- );
229
- ```
230
-
231
- ### Timeouts
232
-
233
- Pass `ctx.timeout` to waits rather than hardcoding numbers, so a slow environment
234
- can be accommodated centrally. A generous timeout on a *correct* condition is
235
- fine — it only ever waits as long as it must, then proceeds the instant the
236
- condition holds. That is the opposite of a fixed sleep.
237
-
238
- ### Prefer robust locators
239
-
240
- Favor role/text/label locators (`getByRole`, `getByText`, `getByLabel`) and stable
241
- attributes (`data-testid`) over brittle CSS/class chains — component-library class
242
- names (`bp6-…`) change between versions. When the only distinguishing feature is
243
- visible text, a text/regex locator is more durable than a guessed class.