patchrome 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +514 -0
- package/bin/patchrome.js +10 -0
- package/dist/build-id.d.ts +2 -0
- package/dist/build-id.js +21 -0
- package/dist/challenges.d.ts +22 -0
- package/dist/challenges.js +97 -0
- package/dist/chrome-profiles.d.ts +17 -0
- package/dist/chrome-profiles.js +141 -0
- package/dist/cli-options.d.ts +131 -0
- package/dist/cli-options.js +43 -0
- package/dist/cli.d.ts +48 -0
- package/dist/cli.js +572 -0
- package/dist/client.d.ts +16 -0
- package/dist/client.js +210 -0
- package/dist/commands.d.ts +58 -0
- package/dist/commands.js +1076 -0
- package/dist/completions.d.ts +1 -0
- package/dist/completions.js +114 -0
- package/dist/copy-guard.d.ts +75 -0
- package/dist/copy-guard.js +167 -0
- package/dist/daemon.d.ts +7 -0
- package/dist/daemon.js +313 -0
- package/dist/diagnostics.d.ts +44 -0
- package/dist/diagnostics.js +117 -0
- package/dist/engine.d.ts +51 -0
- package/dist/engine.js +257 -0
- package/dist/events.d.ts +41 -0
- package/dist/events.js +106 -0
- package/dist/extract.d.ts +27 -0
- package/dist/extract.js +62 -0
- package/dist/focus.d.ts +1 -0
- package/dist/focus.js +44 -0
- package/dist/glob.d.ts +4 -0
- package/dist/glob.js +63 -0
- package/dist/har.d.ts +105 -0
- package/dist/har.js +88 -0
- package/dist/history.d.ts +35 -0
- package/dist/history.js +277 -0
- package/dist/host-platform.d.ts +5 -0
- package/dist/host-platform.js +19 -0
- package/dist/host-prompts-macos.d.ts +2 -0
- package/dist/host-prompts-macos.js +102 -0
- package/dist/host-prompts-wsl.d.ts +6 -0
- package/dist/host-prompts-wsl.js +64 -0
- package/dist/host-prompts.d.ts +3 -0
- package/dist/host-prompts.js +25 -0
- package/dist/index.d.ts +17 -0
- package/dist/index.js +47 -0
- package/dist/network.d.ts +54 -0
- package/dist/network.js +204 -0
- package/dist/origin-storage.d.ts +31 -0
- package/dist/origin-storage.js +82 -0
- package/dist/paths.d.ts +17 -0
- package/dist/paths.js +52 -0
- package/dist/pipe.d.ts +9 -0
- package/dist/pipe.js +73 -0
- package/dist/profile-mode.d.ts +10 -0
- package/dist/profile-mode.js +42 -0
- package/dist/protocol-help.d.ts +34 -0
- package/dist/protocol-help.js +66 -0
- package/dist/protocol.d.ts +49 -0
- package/dist/protocol.js +89 -0
- package/dist/refs.d.ts +9 -0
- package/dist/refs.js +46 -0
- package/dist/routes.d.ts +20 -0
- package/dist/routes.js +106 -0
- package/dist/runner.d.ts +20 -0
- package/dist/runner.js +81 -0
- package/dist/session-name.d.ts +9 -0
- package/dist/session-name.js +50 -0
- package/dist/session-store.d.ts +5 -0
- package/dist/session-store.js +58 -0
- package/dist/sessions.d.ts +47 -0
- package/dist/sessions.js +171 -0
- package/dist/tab-groups.d.ts +9 -0
- package/dist/tab-groups.js +13 -0
- package/dist/targets.d.ts +43 -0
- package/dist/targets.js +229 -0
- package/dist/validate.d.ts +3 -0
- package/dist/validate.js +31 -0
- package/dist/wait.d.ts +24 -0
- package/dist/wait.js +88 -0
- package/examples/go/go.mod +3 -0
- package/examples/go/main.go +104 -0
- package/examples/hn-front-page.sh +18 -0
- package/examples/hn-front-page.ts +24 -0
- package/examples/hn_front_page.py +56 -0
- package/extension/tab-groups/manifest.json +8 -0
- package/extension/tab-groups/service-worker.js +41 -0
- package/package.json +60 -0
- package/skills/patchrome/SKILL.md +74 -0
- package/skills/patchrome/references/commands.md +130 -0
- package/skills/patchrome/references/debugging.md +20 -0
- package/skills/patchrome/references/hard-pages.md +49 -0
- package/skills/patchrome/references/logins.md +46 -0
- package/skills/patchrome/references/scraping.md +51 -0
- package/skills/patchrome/references/scripting.md +79 -0
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Hard pages: iframes, shadow roots, CAPTCHAs, bot walls
|
|
2
|
+
|
|
3
|
+
## iframes
|
|
4
|
+
|
|
5
|
+
Snapshots include iframes, cross-site ones too; their refs carry a different frame prefix, like
|
|
6
|
+
`@f2e4`. Locators need the frame: `click --role checkbox --name Verify --frame 'iframe[title*=widget]'`.
|
|
7
|
+
|
|
8
|
+
## Closed shadow roots
|
|
9
|
+
|
|
10
|
+
A widget inside a closed shadow root shows up empty in the snapshot. CSS selectors pierce it:
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
patchrome click --selector '#cb' --frame 'iframe[src*="challenges"]'
|
|
14
|
+
patchrome fill --selector 'input#code' --frame 'iframe' 123456
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Canvas and anything without an element
|
|
18
|
+
|
|
19
|
+
Take a screenshot, read the pixel position, then:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
patchrome click --at 120,340 # trusted click at viewport pixels, into whatever iframe is under it
|
|
23
|
+
patchrome type "hello" # trusted keystrokes into the focused element
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## CAPTCHAs
|
|
27
|
+
|
|
28
|
+
Never solve one yourself: not by clicking its checkbox, reading its images or typing its characters.
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
patchrome challenge # none | pending | solved, with vendor and box
|
|
32
|
+
patchrome challenge --handoff # raises the tab for the user, waits up to 10 min
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
When `challenge` reports `pending`, tell the user, run `challenge --handoff`, and carry on once it
|
|
36
|
+
prints `solved` or `none`.
|
|
37
|
+
|
|
38
|
+
## Bot-wall interstitials
|
|
39
|
+
|
|
40
|
+
"Just a moment", "Prove your humanity" and similar pages often clear by themselves within seconds. One
|
|
41
|
+
early read reports a block that is not there:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
patchrome open https://www.reddit.com/r/programming/
|
|
45
|
+
patchrome wait --title "Prove your humanity" --gone --timeout-ms 20000
|
|
46
|
+
patchrome snapshot
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Only a wait that times out means blocked. Then run `challenge`, and hand off if it reports `pending`.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Logins and state
|
|
2
|
+
|
|
3
|
+
Cookies and logins are shared by every session in a profile: sign in once and every agent is signed
|
|
4
|
+
in. Never log out of a site another agent may be using.
|
|
5
|
+
|
|
6
|
+
## Pick the way in
|
|
7
|
+
|
|
8
|
+
1. The user says they are signed in to the site in their everyday Chrome: `state import <site>`.
|
|
9
|
+
2. Otherwise, a person signs in: `login <url> --until <url-glob>`.
|
|
10
|
+
3. A saved state file from earlier: `state load <file>`.
|
|
11
|
+
4. A different account, or a clean cookie jar: `open --isolated` as your session's first tab.
|
|
12
|
+
|
|
13
|
+
## `state import`
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
patchrome state import github.com --from "Profile 1"
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
`--from` takes the name in Chrome's profile menu, the folder (`Default`, `Profile 1`) or the signed-in
|
|
20
|
+
email; without it, the last used profile. Ask which profile when the user has several; a wrong
|
|
21
|
+
`--from` lists them. The site covers its subdomains. Google accounts use device-bound sessions and
|
|
22
|
+
may refuse an imported login; use `login` for those.
|
|
23
|
+
|
|
24
|
+
## `login`
|
|
25
|
+
|
|
26
|
+
A visible tab for a person. Tell the user a Chrome window is waiting, and pass `--until` with a URL
|
|
27
|
+
the site reaches after sign-in. Without `--until` it returns when the user closes the tab. It waits
|
|
28
|
+
up to 10 minutes.
|
|
29
|
+
|
|
30
|
+
## Approvals
|
|
31
|
+
|
|
32
|
+
`state import` and `state load` wait up to a minute for the user to approve with Touch ID (Windows
|
|
33
|
+
Hello on WSL). Tell the user a prompt is coming before you run one. `copy_denied` means they refused:
|
|
34
|
+
do not retry, and never copy cookies or profile files another way. `audit` lists past copies.
|
|
35
|
+
|
|
36
|
+
## `state save` and `state load`
|
|
37
|
+
|
|
38
|
+
`state save <file>` writes cookies and localStorage for the sites your session visited, in
|
|
39
|
+
Playwright's storageState format. The file holds live session tokens: keep it out of git.
|
|
40
|
+
`state load <file>` adds to the shared profile without clearing it, so every session sees the cookies.
|
|
41
|
+
|
|
42
|
+
## Isolated sessions
|
|
43
|
+
|
|
44
|
+
`open --isolated` on your session's first tab gives it an in-memory cookie jar. Its cookies vanish
|
|
45
|
+
when the session closes or the daemon restarts, and its tabs are not grouped in Chrome. `state load`
|
|
46
|
+
and `state import` in an isolated session fill only its own jar.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Scraping
|
|
2
|
+
|
|
3
|
+
## Read the API response first
|
|
4
|
+
|
|
5
|
+
Most shops and apps render from JSON. The response the page already fetched beats parsing HTML:
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
patchrome open https://shop.example/product/42 --wait networkidle
|
|
9
|
+
patchrome network list --type xhr,fetch # n17 t2 200 fetch GET https://shop.example/api/product/42 88ms
|
|
10
|
+
patchrome network get n17 --body # JSON inline, or a file path past 2 KB
|
|
11
|
+
patchrome network get --url '*/api/product/*' --body --inline # by URL: the newest match
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
A response body lives only as long as its page. Read it before navigating away, or `tab_gone` follows.
|
|
15
|
+
|
|
16
|
+
## Rows from HTML
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
patchrome extract '{"rows": "li.product", "fields": {"name": "h2", "url": {"selector": "a", "attr": "href"}, "tags": {"selector": ".tag", "all": true}}, "limit": 50}'
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
- A field is a selector string (its visible text), or `{selector, attr, all}`.
|
|
23
|
+
- No `selector` means the row itself. `href` and `src` come back absolute. A missing element is `null`.
|
|
24
|
+
- Pass a file path instead of inline JSON for long schemas.
|
|
25
|
+
- `--role`, `--selector` or `--ref` limits rows to inside one element.
|
|
26
|
+
|
|
27
|
+
## Globs
|
|
28
|
+
|
|
29
|
+
Globs match the whole URL: `*` is any run of characters, `?` one character. `*/api/items` does not
|
|
30
|
+
match `/api/items?page=2`; write `*/api/items*`. Quote them in the shell.
|
|
31
|
+
|
|
32
|
+
## Recording and shaping traffic
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
patchrome network har start
|
|
36
|
+
# ... browse ...
|
|
37
|
+
patchrome network har stop --out traffic.har # your session's requests, with text bodies
|
|
38
|
+
patchrome route block '*/analytics/*'
|
|
39
|
+
patchrome route mock '*/api/flags*' flags.json
|
|
40
|
+
patchrome route clear
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Routes intercept every request on your tabs while any rule exists, which turns off the cache. Clear
|
|
44
|
+
them when done.
|
|
45
|
+
|
|
46
|
+
## Pages that change after load
|
|
47
|
+
|
|
48
|
+
- `click` returns once the click lands; `wait --url` or `wait --text` for the next page before reading.
|
|
49
|
+
- Paginate with a locator: `click --role link --name Next --exact`, then `wait --url '*page=2*'`.
|
|
50
|
+
- Keep volume low on protected sites: one page at a time, no tight `goto` loops.
|
|
51
|
+
- For a scrape the user will rerun, pass `--inline` or `--out <file>` so output keeps one shape.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Scripting a flow
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
- Export what worked
|
|
5
|
+
- Output that keeps its shape
|
|
6
|
+
- `patchrome pipe`
|
|
7
|
+
- The Node library
|
|
8
|
+
- Sessions in scripts
|
|
9
|
+
|
|
10
|
+
## Export what worked
|
|
11
|
+
|
|
12
|
+
Explore with refs, then export the successful commands as a script. The daemon rewrites each ref as
|
|
13
|
+
the role and name from the snapshot it came from.
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
patchrome session history clear # before exploring
|
|
17
|
+
# ... open, snapshot, fill @e5, click @e6, extract ...
|
|
18
|
+
patchrome session history --out scrape.sh # runnable sh
|
|
19
|
+
patchrome session history --format jsonl --out scrape.jsonl
|
|
20
|
+
sh scrape.sh # or: patchrome pipe --bail < scrape.jsonl
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
History leaves out snapshots, tabs, session commands and failed commands. Read every `# check:` line
|
|
24
|
+
before handing the script over:
|
|
25
|
+
|
|
26
|
+
- a repeated role and name got `--nth` from its position on the recorded page
|
|
27
|
+
- a ref inside an iframe needs `--frame <iframe-css>`
|
|
28
|
+
- text typed into a password field became `$PATCHROME_SECRET` (sh) or `<secret>` (jsonl)
|
|
29
|
+
- tab ids and `login`, `challenge --handoff`, `state import` steps need a person or may differ
|
|
30
|
+
|
|
31
|
+
Run the export once in a fresh session (`PATCHROME_SESSION=try-1 sh scrape.sh`) before calling it done.
|
|
32
|
+
|
|
33
|
+
## Output that keeps its shape
|
|
34
|
+
|
|
35
|
+
Without flags, `text`, `eval`, `extract`, `cookies`, `network list` and `network get --body` print a
|
|
36
|
+
value up to 2 KB and a path past that. Scripts pass `--inline` (always the value) or `--out <file>`
|
|
37
|
+
(always the file). With `--json`, values keep their JSON type: `eval` answers `{"value": ...}`,
|
|
38
|
+
`extract` answers `{"count", "rows": [...]}`. Use `network get --url <glob>`, not request ids.
|
|
39
|
+
|
|
40
|
+
## `patchrome pipe`
|
|
41
|
+
|
|
42
|
+
One request per line on stdin: the words after `patchrome` as a JSON array, or
|
|
43
|
+
`{"id": ..., "argv": [...]}`. One response per line on stdout: the `--json` object plus `id` (the line
|
|
44
|
+
number when none was given).
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
printf '%s\n' '["open", "https://news.ycombinator.com"]' \
|
|
48
|
+
'{"id": "top", "argv": ["extract", "{\"rows\": \"tr.athing\", \"fields\": {\"title\": \".titleline > a\"}}", "--inline"]}' \
|
|
49
|
+
| patchrome pipe
|
|
50
|
+
# {"id":1,"ok":true,"data":{"tab":"t1","url":"https://news.ycombinator.com/","title":"Hacker News"}}
|
|
51
|
+
# {"id":"top","ok":true,"data":{"count":30,"rows":[{"title":"..."}]}}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
- Requests for one session run in order; `--session` inside `argv` runs other sessions at once.
|
|
55
|
+
- `watch` and `console --follow` send `{"id", "stream"}` lines first and do not block later requests.
|
|
56
|
+
- `--bail` stops at the first failure. Exit code 0 when every request succeeded, 1 otherwise.
|
|
57
|
+
- Kept open as a coprocess, one pipe serves a whole Python or Go program with no process per step.
|
|
58
|
+
|
|
59
|
+
A Python caller writes `json.dumps({"id": n, "argv": [...]})` to the pipe's stdin and reads lines
|
|
60
|
+
until one has that `id` and no `stream` key. The package's `examples/` folder has sh, Python, Node and
|
|
61
|
+
Go versions.
|
|
62
|
+
|
|
63
|
+
## The Node library
|
|
64
|
+
|
|
65
|
+
```ts
|
|
66
|
+
import { CommandError, connect } from "patchrome";
|
|
67
|
+
|
|
68
|
+
const browser = connect({ session: "prices" });
|
|
69
|
+
await browser.run("open", "https://shop.example");
|
|
70
|
+
const { rows } = await browser.run("extract", "schema.json", "--inline");
|
|
71
|
+
await browser.stream(["watch", "--events", "response", "--count", "3"], (event) => console.log(event));
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
`run` resolves to the `--json` `data` and throws `CommandError` with `code`.
|
|
75
|
+
|
|
76
|
+
## Sessions in scripts
|
|
77
|
+
|
|
78
|
+
A script started from your shell shares your session and tabs. Give each run its own:
|
|
79
|
+
`PATCHROME_SESSION=scrape-$$` in sh, `connect({ session })` in Node. End with `session close`.
|