@only-cli/oc 0.3.0-beta.2 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +46 -18
- package/clis/github.com.json +2 -2
- package/clis/reddit.com.json +1 -1
- package/clis/wikipedia.org.json +8 -0
- package/package.json +1 -1
- package/skills/web-browsing-cli/SKILL.md +29 -10
- package/src/act.js +5 -1
- package/src/cli.js +63 -6
- package/src/distill.js +80 -0
- package/src/fetch.js +332 -28
- package/src/render.js +62 -0
- package/src/sites.js +148 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 only-cli
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|

|
|
4
4
|
|
|
5
|
-
[](https://www.npmjs.com/package/@only-cli/oc) [](https://nodejs.org) [](
|
|
5
|
+
[](https://www.npmjs.com/package/@only-cli/oc) [](https://nodejs.org) [](LICENSE) [](https://scorecard.dev/viewer/?uri=github.com/only-cli/oc)
|
|
6
6
|
|
|
7
7
|
Turns websites into a command line interface for AI agents. `oc open <url>` fetches a page and hands back a compact, numbered view instead of raw HTML or a screenshot, so agents like Claude Code, Codex, and Antigravity can browse without burning tokens. It also gets past blocks that stop naive fetchers on some sites, by talking to the page the way a real browser would.
|
|
8
8
|
|
|
@@ -27,7 +27,7 @@ If you are an LLM reading this repository, [llms.txt](llms.txt) is the short ver
|
|
|
27
27
|
npm install -g @only-cli/oc
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
Requires Node 20+. Requests impersonate Chrome via [impers](https://github.com/lexiforest/impers); falls back to native fetch if impers is unavailable.
|
|
30
|
+
Requires Node 20+. Requests impersonate Chrome via [impers](https://github.com/lexiforest/impers); falls back to native fetch if impers is unavailable. Outbound fetches honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` when set.
|
|
31
31
|
|
|
32
32
|
### Agent skill
|
|
33
33
|
|
|
@@ -64,6 +64,8 @@ oc find <query> where a string appears on the page already open, or
|
|
|
64
64
|
oc read <n> full text of the region at [n], up to 2000 tokens
|
|
65
65
|
oc next the next budget worth of the page already open
|
|
66
66
|
oc raw [url] distilled markdown of the whole page
|
|
67
|
+
oc <site> <verb> ... site shortcut: 'oc hn top', 'oc reddit sub ClaudeAI'
|
|
68
|
+
oc sites the site shortcuts that ship with oc
|
|
67
69
|
oc fill <n> <text> type into a numbered input (planned)
|
|
68
70
|
oc submit [n] submit a form (planned)
|
|
69
71
|
```
|
|
@@ -72,25 +74,30 @@ Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTM
|
|
|
72
74
|
|
|
73
75
|
`oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. A result title on a search page is a link, so `oc do` on it opens the result rather than repeating the title. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page, and a `find` with a single match prints that region instead of the number to read it with. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
|
|
74
76
|
|
|
77
|
+
When a page comes back with no readable text (JavaScript-only, a consent wall, a bot challenge), `oc` says so in one line on stderr and exits 2 instead of printing a title and calling it a render. That is a different exit code from every other failure, and `--json` carries the same verdict as an `empty` field, so an agent can tell "this page has nothing on it" from "oc could not read this page" and pay for a browser only when it is worth it.
|
|
78
|
+
|
|
75
79
|
## Supported websites
|
|
76
80
|
|
|
77
|
-
Works on any mostly-static site with no per-site setup: news sites, blogs, documentation, forums, search engines. A JSON API is a page here too: `oc open` on an endpoint that answers with JSON renders one numbered item per record, keeps the fields that actually differ between items, and says once what every item shares. On top of that, `clis/` ships tuned shortcuts
|
|
81
|
+
Works on any mostly-static site with no per-site setup: news sites, blogs, documentation, forums, search engines. A JSON API is a page here too: `oc open` on an endpoint that answers with JSON renders one numbered item per record, keeps the fields that actually differ between items, and says once what every item shares. On top of that, `clis/` ships tuned shortcuts, so `oc hn item 4711` or `oc gh repo only-cli oc` gets there without the agent knowing how that site spells its URLs. Name the site by its short name, its bare name, or its domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`), and `oc sites` prints the whole list with its verbs:
|
|
78
82
|
|
|
79
|
-
| website |
|
|
83
|
+
| website | command | shortcuts |
|
|
80
84
|
| --- | --- | --- |
|
|
81
|
-
| Hacker News |
|
|
82
|
-
| Reddit | reddit
|
|
83
|
-
| GitHub |
|
|
84
|
-
| X | x
|
|
85
|
-
| LinkedIn | linkedin
|
|
86
|
-
| DuckDuckGo |
|
|
87
|
-
| Bing | bing
|
|
88
|
-
| Stack Overflow |
|
|
89
|
-
| Yahoo Finance |
|
|
90
|
-
| YouTube |
|
|
91
|
-
|
|
|
92
|
-
|
|
|
93
|
-
|
|
|
85
|
+
| Hacker News | `oc hn` | `top`, `new`, `item <id>`, `user <name>` |
|
|
86
|
+
| Reddit | `oc reddit` (via old.reddit.com) | `sub <name>`, `post <id>`, `user <name>`, `search <query>` |
|
|
87
|
+
| GitHub | `oc gh` | `repo <owner> <name>`, `user <name>`, `search <query>`, `trending`, `issues <owner> <name>` |
|
|
88
|
+
| X | `oc x` | `user <name>`, `post <id>` |
|
|
89
|
+
| LinkedIn | `oc linkedin` | `profile <name>`, `company <name>`, `jobs <query>` (public guest views) |
|
|
90
|
+
| DuckDuckGo | `oc ddg` | `search <query>`, `lite <query>` |
|
|
91
|
+
| Bing | `oc bing` | `search <query>`, `news <query>` |
|
|
92
|
+
| Stack Overflow | `oc so` (via Atom feeds and the Stack Exchange API) | `search <query>`, `question <id>`, `tag <name>`, `user <id>`, `recent` |
|
|
93
|
+
| Yahoo Finance | `oc yahoo` | `quote <symbol>`, `news <symbol>`, `history <symbol>`, `lookup <query>`, `markets`, `gainers`, `losers`, `trending` |
|
|
94
|
+
| YouTube | `oc yt` | `video <id>`, `channel <name>` |
|
|
95
|
+
| Wikipedia | `oc wiki` (via `action=render`) | `article <title>`, `search <query>`, `lang <code> <title>` |
|
|
96
|
+
| AWS docs | `oc aws` (search via DuckDuckGo) | `guide <service> <page>`, `page <service> <guide> <page>`, `cli <command>`, `search <query>` |
|
|
97
|
+
| Google Cloud docs | `oc gcp` (via docs.cloud.google.com, search via DuckDuckGo) | `docs <product>`, `page <product> <page>`, `gcloud <command>`, `search <query>` |
|
|
98
|
+
| Microsoft Learn | `oc learn` (search via its RSS API) | `azure <page>`, `doc <path>`, `cli <command>`, `search <query>` |
|
|
99
|
+
|
|
100
|
+
A shortcut only ever resolves to a URL and then takes the same path `oc open` does, so it changes nothing about what a page costs or how it reads. The last argument takes every word after it, so `oc ddg search claude code cli` and `oc aws search s3 lifecycle rules` need no quoting, and a path argument keeps its slashes, so `oc learn doc azure/aks/what-is-aks` reaches that page.
|
|
94
101
|
|
|
95
102
|
A few of these (X, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. AWS and Google Cloud render docs search purely client-side with no feed, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead. Not supported yet: pages that only render with JavaScript, sites behind logins, and sites with hard bot challenges that expose no feed.
|
|
96
103
|
|
|
@@ -106,6 +113,27 @@ Full methodology, per-task numbers, and other agents/models live in [only-cli/be
|
|
|
106
113
|
| Jina Reader | 16,402 | blocked on the Reddit page |
|
|
107
114
|
| raw HTML fetch | 177,685 | blocked on the search page |
|
|
108
115
|
|
|
116
|
+
Read cost is one thing, but what an agent actually spends is another, so a
|
|
117
|
+
second suite runs whole tasks end to end in Claude Code and compares `oc`
|
|
118
|
+
against the tools the agent already has. Five Wikipedia lookups, one tool per
|
|
119
|
+
run, Sonnet driving:
|
|
120
|
+
|
|
121
|
+
| tool | answered correctly | input tokens | cost | turns | avg time |
|
|
122
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
123
|
+
| `oc wiki` | 5/5 | 5,535 | $0.27 | 22 | 11s |
|
|
124
|
+
| built-in `WebFetch` | 5/5 | 128,792 | $0.37 | 25 | 14s |
|
|
125
|
+
| built-in `WebSearch` | 5/5 | 160,431 | $0.52 | 27 | 22s |
|
|
126
|
+
|
|
127
|
+
All three got every answer right, so this is a cost result, not an accuracy one.
|
|
128
|
+
Input tokens are the fresh context each tool put in front of the model, which is
|
|
129
|
+
the number the page size drives; totals including cache reads sit closer together
|
|
130
|
+
because the agent's own prompt dominates them. The spread widens with the page:
|
|
131
|
+
`oc` cost 5.7x less than `WebFetch` on a short stub and 35x less on a long
|
|
132
|
+
article, because the 500 token budget makes it flat at about 1,100 tokens per
|
|
133
|
+
page while a full fetch pays for whatever the page weighs. `WebSearch` was given
|
|
134
|
+
only the question, not the article URL, which is the honest way to use it and
|
|
135
|
+
part of why it costs the most.
|
|
136
|
+
|
|
109
137
|
## Status
|
|
110
138
|
|
|
111
139
|
Early. Reading works and is covered by offline tests: static pages, XML feeds, JSON APIs, budget-aware rendering, sessions, and the numbered actions `do`, `find`, `read`, `next`, and `raw`. Writing does not: `fill`, `submit`, and `back` report that they are not implemented rather than pretending, and a lazy headless fallback for script-heavy pages comes after them.
|
|
@@ -118,4 +146,4 @@ Known limits, honestly: no JavaScript rendering yet, no sites behind logins yet,
|
|
|
118
146
|
|
|
119
147
|
## License
|
|
120
148
|
|
|
121
|
-
MIT
|
|
149
|
+
MIT, see [LICENSE](LICENSE).
|
package/clis/github.com.json
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
{
|
|
2
2
|
"domain": "github.com",
|
|
3
3
|
"commands": {
|
|
4
|
-
"repo": { "open": "https://github.com/{owner}/{
|
|
4
|
+
"repo": { "open": "https://github.com/{owner}/{name}", "args": ["owner", "name"] },
|
|
5
5
|
"user": { "open": "https://github.com/{name}", "args": ["name"] },
|
|
6
6
|
"search": { "open": "https://github.com/search?q={query}&type=repositories", "args": ["query"] },
|
|
7
7
|
"trending": { "open": "https://github.com/trending" },
|
|
8
|
-
"issues": { "open": "https://github.com/{owner}/{
|
|
8
|
+
"issues": { "open": "https://github.com/{owner}/{name}/issues", "args": ["owner", "name"] }
|
|
9
9
|
}
|
|
10
10
|
}
|
package/clis/reddit.com.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"domain": "reddit.com",
|
|
3
3
|
"commands": {
|
|
4
|
-
"sub": { "open": "https://old.reddit.com/r/{
|
|
4
|
+
"sub": { "open": "https://old.reddit.com/r/{name}", "args": ["name"] },
|
|
5
5
|
"post": { "open": "https://old.reddit.com/comments/{id}", "args": ["id"] },
|
|
6
6
|
"user": { "open": "https://old.reddit.com/user/{name}", "args": ["name"] },
|
|
7
7
|
"search": { "open": "https://old.reddit.com/search?q={query}", "args": ["query"] }
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
{
|
|
2
|
+
"domain": "wikipedia.org",
|
|
3
|
+
"commands": {
|
|
4
|
+
"article": { "open": "https://en.wikipedia.org/w/index.php?title={title}&action=render", "args": ["title"] },
|
|
5
|
+
"search": { "open": "https://en.wikipedia.org/w/index.php?search={query}&fulltext=1&ns0=1", "args": ["query"] },
|
|
6
|
+
"lang": { "open": "https://{code}.wikipedia.org/w/index.php?title={title}&action=render", "args": ["code", "title"] }
|
|
7
|
+
}
|
|
8
|
+
}
|
package/package.json
CHANGED
|
@@ -8,28 +8,45 @@ description: Token-efficient web browsing and web content extraction for AI agen
|
|
|
8
8
|
Renders a web page as a compact, numbered terminal view instead of raw HTML. A typical page is under 500 tokens.
|
|
9
9
|
|
|
10
10
|
```
|
|
11
|
-
npx --yes @only-cli/oc@0.
|
|
12
|
-
npx --yes @only-cli/oc@0.
|
|
13
|
-
npx --yes @only-cli/oc@0.
|
|
14
|
-
|
|
15
|
-
npx --yes @only-cli/oc@0.
|
|
16
|
-
npx --yes @only-cli/oc@0.
|
|
11
|
+
npx --yes @only-cli/oc@0.4.0 open <url> compact view, numbered elements
|
|
12
|
+
npx --yes @only-cli/oc@0.4.0 do <n> follow link [n], or read it if [n] is text
|
|
13
|
+
npx --yes @only-cli/oc@0.4.0 find <query> where a string appears, or that place itself
|
|
14
|
+
when only one matches
|
|
15
|
+
npx --yes @only-cli/oc@0.4.0 next next ~500 tokens of the page already open
|
|
16
|
+
npx --yes @only-cli/oc@0.4.0 read <n> full text of region [n]
|
|
17
|
+
npx --yes @only-cli/oc@0.4.0 raw [url] whole page as markdown (--html for cleaned HTML)
|
|
17
18
|
```
|
|
18
19
|
|
|
19
20
|
None of these except `open`/`do`/`raw <url>` fetch anything — they replay the page `open` already saved.
|
|
20
21
|
|
|
22
|
+
## Site shortcuts
|
|
23
|
+
|
|
24
|
+
`oc <site> <verb> [args]` resolves to a URL and then behaves exactly like `open` on it, so it costs the same and reads the same. It saves guessing a URL shape and, on a few sites, points at the feed or public API that answers without a login.
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
oc hn top oc reddit sub ClaudeAI oc gh repo only-cli oc
|
|
28
|
+
oc wiki article Eiffel Tower oc wiki search anthropic oc wiki lang de Berlin
|
|
29
|
+
oc ddg search claude code oc so question 231767 oc learn doc azure/aks/what-is-aks
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Sites: `hn`, `reddit`, `gh`, `x`, `linkedin`, `ddg`, `bing`, `so`, `finance`, `yt`, `aws`, `gcp`, `learn`, `wiki`. Name one by short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`). The last argument takes every word after it, so a query or title needs no quoting. `oc sites` lists every site with its verbs, which is cheaper than guessing one.
|
|
33
|
+
|
|
34
|
+
Prefer a shortcut over a hand-built URL when one exists for the site, and prefer `oc wiki article <title>` over a search when you already know the article's name.
|
|
35
|
+
|
|
36
|
+
|
|
21
37
|
## Output
|
|
22
38
|
|
|
23
39
|
- Line 1 is the title, then main content (article/thread/results); nav/sidebar/footer follow after `--- rest of page ---`, still numbered.
|
|
24
40
|
- `--- repeated controls hidden ---`: per-item chrome (save/report/reply) dropped as repetitive; `raw` keeps it.
|
|
25
41
|
- `[n]` marks a link, button, input, heading, or a text block long enough to be cut.
|
|
26
|
-
-
|
|
42
|
+
- Code blocks arrive as the page wrote them, lines and indentation intact, so a command in one can be run as printed.
|
|
43
|
+
- `... +820 chars`: block was cut there; `read <n>` prints it whole. The cut lands on the end of a sentence, or of a line in code, so what is shown is never half of one.
|
|
27
44
|
- `... 164 more blocks (~7,100 tokens)`: rest of page past budget — a cost estimate, not a fetch. Omitted when the page would finish only a little over budget; then it's printed whole instead.
|
|
28
45
|
- `actions:` footer lists valid next commands.
|
|
29
46
|
|
|
30
47
|
## Going further, cheapest first
|
|
31
48
|
|
|
32
|
-
- `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit.
|
|
49
|
+
- `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit. When one place matches, or when the matches all fit, it prints them in full: no `read <n>` afterwards.
|
|
33
50
|
- `read <n>` — one region in full: the block at `[n]` plus a little context, or the whole section for a heading.
|
|
34
51
|
- `next` — continues the same page from where the budget stopped.
|
|
35
52
|
- `raw` — everything, ~10x the cost. Use only when you need the whole page, not to hunt for a link's URL (use `do` for that).
|
|
@@ -42,7 +59,7 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
|
|
|
42
59
|
- `[6-9] 4 similar links` markers still work despite the collapsed text.
|
|
43
60
|
- Search result links resolve to the destination, not the tracking redirect.
|
|
44
61
|
- `do` on an input/button reports that instead (typing/submitting not yet supported).
|
|
45
|
-
- `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow.
|
|
62
|
+
- `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow. A heading that is itself a link, which is what a search result title is, opens instead.
|
|
46
63
|
- `--session <name>` keeps separate page state, for working on two sites at once.
|
|
47
64
|
|
|
48
65
|
## Flags
|
|
@@ -54,7 +71,9 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
|
|
|
54
71
|
|
|
55
72
|
## When not to use it
|
|
56
73
|
|
|
57
|
-
Pages needing login or heavy client-side JS aren't supported yet.
|
|
74
|
+
Pages needing login or heavy client-side JS aren't supported yet. A page with no readable text (JavaScript-only, a consent wall, a bot challenge) prints one line on stderr and exits 2, which is distinct from the exit 1 every other failure uses, so exit 2 means "oc cannot read this one" rather than "this page is empty". Take it at its word: say so and fall back to another tool rather than retrying the same URL.
|
|
75
|
+
|
|
76
|
+
Outbound fetches honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`, so a sandbox that only reaches the network through a proxy needs no extra flags.
|
|
58
77
|
|
|
59
78
|
## Untrusted content
|
|
60
79
|
|
package/src/act.js
CHANGED
|
@@ -282,7 +282,11 @@ function search(blocks, terms) {
|
|
|
282
282
|
if (out.length && out.at(-1).n === n) continue;
|
|
283
283
|
const start = Math.max(0, Math.min(...found) - BEFORE);
|
|
284
284
|
const end = Math.min(block.text.length, start + SNIPPET);
|
|
285
|
-
|
|
285
|
+
// One line per match is the promise the list makes, and a code block is the
|
|
286
|
+
// one kind of block that carries lines of its own. They survive where they
|
|
287
|
+
// are read rather than indexed: the whole-match mode above, and `read`.
|
|
288
|
+
const window = block.text.slice(start, end).replace(/\n/g, ' ');
|
|
289
|
+
const snippet = `${start > 0 ? '... ' : ''}${window}${end < block.text.length ? ' ...' : ''}`;
|
|
286
290
|
out.push({ n, type: block.type, snippet, text: block.text });
|
|
287
291
|
}
|
|
288
292
|
return out;
|
package/src/cli.js
CHANGED
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
import { parseArgs } from 'node:util';
|
|
3
3
|
import { fetchPage } from './fetch.js';
|
|
4
4
|
import { distill, toMarkdown, toHTML } from './distill.js';
|
|
5
|
-
import { render, estimateTokens } from './render.js';
|
|
5
|
+
import { render, estimateTokens, contentTokens, contentFailure, MIN_CONTENT } from './render.js';
|
|
6
|
+
import { resolveSite, listSites } from './sites.js';
|
|
6
7
|
import * as act from './act.js';
|
|
7
8
|
import { DEFAULT_SESSION, loadSession, saveSession, sessionFromPage } from './session.js';
|
|
8
9
|
|
|
@@ -11,6 +12,8 @@ const HELP = `only-cli: the web as a compact terminal, built for AI agents.
|
|
|
11
12
|
usage: oc <command> [args] [flags]
|
|
12
13
|
|
|
13
14
|
open <url> fetch and render a page with numbered actions
|
|
15
|
+
<site> <verb> ... site shortcut: 'oc hn top', 'oc reddit sub ClaudeAI'
|
|
16
|
+
sites the site shortcuts that ship with oc
|
|
14
17
|
find <query> where a string appears on the page already open, or
|
|
15
18
|
the region itself when only one place matches
|
|
16
19
|
next the next budget worth of the page already open
|
|
@@ -35,6 +38,10 @@ flags:
|
|
|
35
38
|
globally. Off by default because metrics cost tokens too.
|
|
36
39
|
--session <name> keep separate page state under a name (default: default)
|
|
37
40
|
|
|
41
|
+
A page that comes back with no readable text (JavaScript-only, a consent wall,
|
|
42
|
+
a bot challenge) says so in one line on stderr and exits 2, so a caller can
|
|
43
|
+
tell an empty page from a page oc could not read and fall back to a browser.
|
|
44
|
+
|
|
38
45
|
'oc open' remembers the page it printed, so 'oc do 3' follows link [3] without
|
|
39
46
|
you ever handling its URL, and 'oc next' or 'oc read 12' picks up what the
|
|
40
47
|
budget left behind without fetching it again. 'oc do' on a search result title
|
|
@@ -60,6 +67,23 @@ const remember = (page, name, cursor) => {
|
|
|
60
67
|
const savings = (out, raw) =>
|
|
61
68
|
`~${out} tokens vs ~${raw} for the page HTML (${Math.max(0, 100 - Math.round((out / Math.max(raw, 1)) * 100))}% saved)`;
|
|
62
69
|
|
|
70
|
+
// Nonzero, and distinct from the exit 1 that every other failure uses, so a
|
|
71
|
+
// caller can branch on 'oc could not read this' without parsing prose.
|
|
72
|
+
const NO_CONTENT_EXIT = 2;
|
|
73
|
+
|
|
74
|
+
const noContent = (url, detail, hint = "; 'oc raw' has the page's markdown if there is any, otherwise this one needs a browser") => {
|
|
75
|
+
console.error(`oc: no readable content at ${url} (${detail}), so it is JavaScript-only, gated, or challenged${hint}`);
|
|
76
|
+
// Not process.exit: stdout may still be draining, and whatever did render is
|
|
77
|
+
// worth printing even when the render failed.
|
|
78
|
+
process.exitCode = NO_CONTENT_EXIT;
|
|
79
|
+
};
|
|
80
|
+
|
|
81
|
+
// Anything else in the first position is tried as a site shortcut before it is
|
|
82
|
+
// called unknown, so a new clis/ definition needs no change here.
|
|
83
|
+
const COMMANDS = new Set([
|
|
84
|
+
'open', 'do', 'raw', 'read', 'next', 'find', 'fill', 'submit', 'back', 'session', 'sites',
|
|
85
|
+
]);
|
|
86
|
+
|
|
63
87
|
async function main() {
|
|
64
88
|
const { values, positionals } = parseArgs({
|
|
65
89
|
allowPositionals: true,
|
|
@@ -78,12 +102,22 @@ async function main() {
|
|
|
78
102
|
// when their own verbose mode is on, or the user exports OC_VERBOSE=1.
|
|
79
103
|
const verbose = values.stats || values.verbose || process.env.OC_VERBOSE === '1';
|
|
80
104
|
|
|
81
|
-
|
|
105
|
+
let [command, ...args] = positionals;
|
|
82
106
|
if (values.help || !command) {
|
|
83
107
|
console.log(HELP);
|
|
84
108
|
return;
|
|
85
109
|
}
|
|
86
110
|
|
|
111
|
+
// A first word that is not a command may still be a site oc ships a
|
|
112
|
+
// definition for, and a shortcut is only ever a URL, so it resolves to one
|
|
113
|
+
// here and the rest of this function never learns it was not typed.
|
|
114
|
+
if (!COMMANDS.has(command)) {
|
|
115
|
+
const site = resolveSite(command, args);
|
|
116
|
+
if (!site) throw new Error(`unknown command '${command}', run oc --help`);
|
|
117
|
+
args = [site.url];
|
|
118
|
+
command = 'open';
|
|
119
|
+
}
|
|
120
|
+
|
|
87
121
|
const sessionName = values.session || DEFAULT_SESSION;
|
|
88
122
|
// Zero means "whatever this command's default is", which differs: the
|
|
89
123
|
// compact view targets 500 tokens, read targets 2000.
|
|
@@ -123,27 +157,49 @@ async function main() {
|
|
|
123
157
|
return `HTTP ${status} via ${via}, fetch ${Math.round(fetchMs)}ms, process ${Math.round(processMs)}ms, `
|
|
124
158
|
+ `${Math.round(html.length / 1024)}KB transferred, ${Math.round(rss / 1048576)}MB memory`;
|
|
125
159
|
};
|
|
160
|
+
const htmlTokens = estimateTokens(html);
|
|
126
161
|
if (values.json) {
|
|
127
162
|
const page = distill(html, finalUrl);
|
|
128
163
|
remember(page, sessionName);
|
|
129
|
-
|
|
164
|
+
const failure = contentFailure(contentTokens(page), htmlTokens);
|
|
165
|
+
// Always present, so a caller can branch on the field rather than on
|
|
166
|
+
// whether a field it was hoping for turned up.
|
|
167
|
+
console.log(JSON.stringify({ ...page, empty: failure != null }));
|
|
130
168
|
if (verbose) console.error(resources());
|
|
169
|
+
if (failure) noContent(finalUrl, failure);
|
|
131
170
|
return;
|
|
132
171
|
}
|
|
133
|
-
const htmlTokens = estimateTokens(html);
|
|
134
172
|
if (command === 'raw') {
|
|
135
173
|
const out = values.html ? toHTML(html, finalUrl) : toMarkdown(html, finalUrl);
|
|
174
|
+
const outTokens = estimateTokens(out);
|
|
136
175
|
console.log(out);
|
|
137
|
-
if (verbose)
|
|
176
|
+
if (verbose) {
|
|
177
|
+
const cost = outTokens < MIN_CONTENT
|
|
178
|
+
? `nothing distilled out of ~${htmlTokens} tokens of page HTML`
|
|
179
|
+
: savings(outTokens, htmlTokens);
|
|
180
|
+
console.error(`${cost}; ${resources()}`);
|
|
181
|
+
}
|
|
182
|
+
// Only the blank case here. `raw` is the fallback the compact view's
|
|
183
|
+
// failure line names, so it must not fail on the same pages: a page
|
|
184
|
+
// whose only text is its menu still has markup, and printing it is the
|
|
185
|
+
// whole point of `raw`.
|
|
186
|
+
if (outTokens < MIN_CONTENT) noContent(finalUrl, `~${outTokens} tokens of markdown`, '');
|
|
138
187
|
return;
|
|
139
188
|
}
|
|
140
189
|
const page = distill(html, finalUrl);
|
|
190
|
+
const failure = contentFailure(contentTokens(page), htmlTokens);
|
|
141
191
|
const { text, stats } = render(page, { budget });
|
|
142
192
|
remember(page, sessionName, stats.next);
|
|
143
193
|
console.log(text);
|
|
144
194
|
if (verbose) {
|
|
145
|
-
|
|
195
|
+
// Reporting '100% saved' of a render that extracted nothing is the one
|
|
196
|
+
// place this line lies, and it lies in the tool's own favour.
|
|
197
|
+
const cost = failure
|
|
198
|
+
? `no content distilled out of ~${htmlTokens} tokens of page HTML`
|
|
199
|
+
: savings(stats.tokens, htmlTokens);
|
|
200
|
+
console.error(`~${stats.tokens} tokens, ${stats.rendered}/${stats.blocks} blocks rendered, ${cost}; ${resources()}`);
|
|
146
201
|
}
|
|
202
|
+
if (failure) noContent(finalUrl, failure);
|
|
147
203
|
return;
|
|
148
204
|
}
|
|
149
205
|
case 'read': return console.log(act.read(Number(args[0]), { session: sessionName, budget: asked || 2000 }));
|
|
@@ -152,6 +208,7 @@ async function main() {
|
|
|
152
208
|
case 'fill': return act.fill(Number(args[0]), args.slice(1).join(' '));
|
|
153
209
|
case 'submit': return act.submit(args[0] ? Number(args[0]) : undefined);
|
|
154
210
|
case 'back': return act.back();
|
|
211
|
+
case 'sites': return console.log(listSites());
|
|
155
212
|
case 'session': throw new act.NotImplemented('session');
|
|
156
213
|
default:
|
|
157
214
|
throw new Error(`unknown command '${command}', run oc --help`);
|
package/src/distill.js
CHANGED
|
@@ -32,6 +32,32 @@ const DROP = new Set([
|
|
|
32
32
|
// them, so the search below refuses to descend into them.
|
|
33
33
|
const FURNITURE = new Set(['nav', 'header', 'footer', 'aside']);
|
|
34
34
|
|
|
35
|
+
// Controls a page puts inside its own code samples, and the selector that
|
|
36
|
+
// finds one. Node's API docs give every code block a copy button, a module
|
|
37
|
+
// toggle and a language label, so the text of the sample ends in
|
|
38
|
+
// `javascriptcopy` unless the toolbar holding them is left out of it.
|
|
39
|
+
const CODE_CONTROLS = new Set(['button', 'input', 'select', 'textarea', 'label']);
|
|
40
|
+
const CONTROL = [...CODE_CONTROLS].join(', ');
|
|
41
|
+
|
|
42
|
+
// Lines in a code block are kept, where every other kind of text has its
|
|
43
|
+
// whitespace collapsed. Joining them would put `// comment` in front of the
|
|
44
|
+
// statements that followed it, so a sample an agent could run would arrive
|
|
45
|
+
// commented out, and the shape of the wreck is invisible on one line. Blank
|
|
46
|
+
// runs and the indentation the whole block shares are the parts that carry no
|
|
47
|
+
// meaning, so those go.
|
|
48
|
+
const codeText = (raw) => {
|
|
49
|
+
const lines = raw.replace(/\r\n?/g, '\n').replace(/[^\S\n]+$/gm, '').split('\n');
|
|
50
|
+
while (lines.length && !lines[0].trim()) lines.shift();
|
|
51
|
+
while (lines.length && !lines[lines.length - 1].trim()) lines.pop();
|
|
52
|
+
const indent = lines
|
|
53
|
+
.filter((l) => l.trim())
|
|
54
|
+
.reduce((least, l) => Math.min(least, l.length - l.trimStart().length), Infinity);
|
|
55
|
+
return lines
|
|
56
|
+
.map((l) => (Number.isFinite(indent) ? l.slice(indent) : l).trimEnd())
|
|
57
|
+
.join('\n')
|
|
58
|
+
.replace(/\n{3,}/g, '\n\n');
|
|
59
|
+
};
|
|
60
|
+
|
|
35
61
|
// Elements that end a line on the page and so must end one here. Without this
|
|
36
62
|
// the text of six separate posts merges into a single block, because nothing
|
|
37
63
|
// between them survives distillation to keep them apart.
|
|
@@ -128,6 +154,45 @@ export function distill(html, url = '') {
|
|
|
128
154
|
/** Subtree already emitted, skipped when the rest of the page is walked. */
|
|
129
155
|
let done = null;
|
|
130
156
|
|
|
157
|
+
/**
|
|
158
|
+
* The text of a code subtree, read as one string.
|
|
159
|
+
*
|
|
160
|
+
* A syntax highlighter gives every token its own element, so `s3://bucket/`
|
|
161
|
+
* reaches the walk as `s3`, `:`, `//`, `bucket`, `/`, and the rule that puts
|
|
162
|
+
* fragments back together only glues the ones that share a parent. The rest
|
|
163
|
+
* are space-joined, which turned an AWS example into
|
|
164
|
+
* `aws s3 cp s3 : // bucket / -- recursive`, a command an agent cannot run.
|
|
165
|
+
* Reading the subtree whole is what fixes that, and it discards nothing: over
|
|
166
|
+
* 172 pre elements on the AWS CLI reference, the Rust book, the Node API docs
|
|
167
|
+
* and the Python library docs, 159 are split this way and not one of them
|
|
168
|
+
* contains a link.
|
|
169
|
+
*
|
|
170
|
+
* textContent would be enough if pages put only code in their code blocks.
|
|
171
|
+
* A control inside one is chrome, and so is the block-level element holding
|
|
172
|
+
* it: that is how the toolbar's `javascript` label leaves with the copy
|
|
173
|
+
* button it sits beside. The test stays on block-level wrappers because a
|
|
174
|
+
* highlighter's own elements are inline, so a stray control can never take a
|
|
175
|
+
* line of code out with it.
|
|
176
|
+
* @param {any} node
|
|
177
|
+
* @returns {string}
|
|
178
|
+
*/
|
|
179
|
+
const verbatim = (node) => {
|
|
180
|
+
let out = '';
|
|
181
|
+
const gather = (n) => {
|
|
182
|
+
if (n.nodeType === 3) {
|
|
183
|
+
out += n.textContent ?? '';
|
|
184
|
+
return;
|
|
185
|
+
}
|
|
186
|
+
if (n.nodeType !== 1) return;
|
|
187
|
+
const tag = n.localName;
|
|
188
|
+
if (DROP.has(tag) || CODE_CONTROLS.has(tag) || hidden(n)) return;
|
|
189
|
+
if (n !== node && BLOCKY.has(tag) && n.querySelector(CONTROL)) return;
|
|
190
|
+
for (const child of n.childNodes) gather(child);
|
|
191
|
+
};
|
|
192
|
+
gather(node);
|
|
193
|
+
return out;
|
|
194
|
+
};
|
|
195
|
+
|
|
131
196
|
const walk = (node) => {
|
|
132
197
|
if (node.nodeType === 3) {
|
|
133
198
|
const raw = node.textContent ?? '';
|
|
@@ -183,6 +248,21 @@ export function distill(html, url = '') {
|
|
|
183
248
|
if (text) blocks.push({ type: 'button', text });
|
|
184
249
|
return;
|
|
185
250
|
}
|
|
251
|
+
if (tag === 'pre' || tag === 'code') {
|
|
252
|
+
const raw = verbatim(node);
|
|
253
|
+
const text = tag === 'pre' ? codeText(raw) : clean(raw);
|
|
254
|
+
// A pre is its own line, which is what BLOCKY gave it before this branch
|
|
255
|
+
// started claiming it first. Inline code belongs to the sentence around
|
|
256
|
+
// it, so it carries the parent and the edge spacing a text node would and
|
|
257
|
+
// merges back into that sentence the same way.
|
|
258
|
+
const own = tag === 'pre';
|
|
259
|
+
if (own) blocks.push({ type: 'break' });
|
|
260
|
+
if (text) {
|
|
261
|
+
blocks.push({ type: 'text', text, host: node.parentNode, pre: /^\s/.test(raw), post: /\s$/.test(raw) });
|
|
262
|
+
}
|
|
263
|
+
if (own) blocks.push({ type: 'break' });
|
|
264
|
+
return;
|
|
265
|
+
}
|
|
186
266
|
if (BLOCKY.has(tag)) {
|
|
187
267
|
blocks.push({ type: 'break' });
|
|
188
268
|
for (const child of node.childNodes) walk(child);
|
package/src/fetch.js
CHANGED
|
@@ -8,7 +8,10 @@
|
|
|
8
8
|
*/
|
|
9
9
|
|
|
10
10
|
import dns from 'node:dns/promises';
|
|
11
|
+
import http from 'node:http';
|
|
12
|
+
import https from 'node:https';
|
|
11
13
|
import net from 'node:net';
|
|
14
|
+
import tls from 'node:tls';
|
|
12
15
|
|
|
13
16
|
// The fetch fallback can't fake a TLS fingerprint like impers does, but it
|
|
14
17
|
// should at least send the same Chrome identity in its headers.
|
|
@@ -23,6 +26,8 @@ const loadImpers = () => {
|
|
|
23
26
|
|
|
24
27
|
const BLOCKED_MESSAGE = 'blocked: private or internal URL';
|
|
25
28
|
const MAX_REDIRECTS = 20;
|
|
29
|
+
// Match undici's default so proxy transport does not hang indefinitely.
|
|
30
|
+
const PROXY_TIMEOUT_MS = 300_000;
|
|
26
31
|
|
|
27
32
|
// What oc can turn into text: any text/* type, plus the application/* types
|
|
28
33
|
// that are really text (json, xml, and the +json / +xml families a feed or an
|
|
@@ -139,13 +144,304 @@ async function assertSafeTarget(urlStr) {
|
|
|
139
144
|
return;
|
|
140
145
|
}
|
|
141
146
|
if (hostname === 'localhost') throw new Error(BLOCKED_MESSAGE);
|
|
142
|
-
const addresses = await dns.lookup(hostname, { all: true }).catch(() =>
|
|
147
|
+
const addresses = await dns.lookup(hostname, { all: true }).catch(() => {
|
|
148
|
+
// With a proxy the client never resolves the target; the proxy does, often
|
|
149
|
+
// on corporate DNS where internal names are NXDOMAIN locally. Fail closed.
|
|
150
|
+
if (resolveProxy(urlStr)) throw new Error(BLOCKED_MESSAGE);
|
|
151
|
+
return [];
|
|
152
|
+
});
|
|
143
153
|
for (const { address, family } of addresses) {
|
|
144
154
|
if (family === 4 && isBlockedIPv4(address)) throw new Error(BLOCKED_MESSAGE);
|
|
145
155
|
if (family === 6 && isBlockedIPv6(address)) throw new Error(BLOCKED_MESSAGE);
|
|
146
156
|
}
|
|
147
157
|
}
|
|
148
158
|
|
|
159
|
+
function envFirst(env, ...names) {
|
|
160
|
+
for (const name of names) {
|
|
161
|
+
const value = env[name];
|
|
162
|
+
if (value) return value;
|
|
163
|
+
}
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
function normalizeProxy(value) {
|
|
167
|
+
if (value == null) return null;
|
|
168
|
+
const trimmed = String(value).trim();
|
|
169
|
+
if (!trimmed) return null;
|
|
170
|
+
return /^[a-z][a-z0-9+.-]*:\/\//i.test(trimmed) ? trimmed : `http://${trimmed}`;
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
function assertHttpProxyProtocol(proxyUrl) {
|
|
174
|
+
if (!/^https?:$/.test(proxyUrl.protocol)) {
|
|
175
|
+
throw new Error(`unsupported proxy protocol (${proxyUrl.protocol.slice(0, -1)}), oc honors HTTP and HTTPS proxies`);
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
// Split host[:port], including [IPv6]:port. Unbracketed IPv6 literals never
|
|
180
|
+
// carry a port suffix (use [addr]:port); a trailing :digits on ::1 is part of
|
|
181
|
+
// the address, not a port.
|
|
182
|
+
function splitHostPort(entry) {
|
|
183
|
+
if (entry.startsWith('[')) {
|
|
184
|
+
const end = entry.indexOf(']');
|
|
185
|
+
if (end === -1) return { host: entry, port: '' };
|
|
186
|
+
const rest = entry.slice(end + 1);
|
|
187
|
+
return { host: entry.slice(1, end), port: rest.startsWith(':') ? rest.slice(1) : '' };
|
|
188
|
+
}
|
|
189
|
+
if (net.isIP(entry) === 6) return { host: entry, port: '' };
|
|
190
|
+
const colon = entry.lastIndexOf(':');
|
|
191
|
+
if (colon !== -1 && /^\d+$/.test(entry.slice(colon + 1))) {
|
|
192
|
+
const host = entry.slice(0, colon);
|
|
193
|
+
if (net.isIP(host) === 6) return { host: entry, port: '' };
|
|
194
|
+
return { host, port: entry.slice(colon + 1) };
|
|
195
|
+
}
|
|
196
|
+
return { host: entry, port: '' };
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
function ipv4InCidr(ip, base, bits) {
|
|
200
|
+
const mask = bits === 0 ? 0 : (0xffffffff << (32 - bits)) >>> 0;
|
|
201
|
+
return (ipv4ToInt(ip) & mask) === (ipv4ToInt(base) & mask);
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
function ipv6InCidr(ip, base, bits) {
|
|
205
|
+
const g = expandIPv6(ip);
|
|
206
|
+
const b = expandIPv6(base);
|
|
207
|
+
if (!g || !b) return false;
|
|
208
|
+
let remaining = bits;
|
|
209
|
+
for (let i = 0; i < 8 && remaining > 0; i++) {
|
|
210
|
+
if (remaining >= 16) {
|
|
211
|
+
if (g[i] !== b[i]) return false;
|
|
212
|
+
remaining -= 16;
|
|
213
|
+
} else {
|
|
214
|
+
const mask = (0xffff << (16 - remaining)) & 0xffff;
|
|
215
|
+
if ((g[i] & mask) !== (b[i] & mask)) return false;
|
|
216
|
+
remaining = 0;
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
return true;
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
function ipInCidr(ip, base, bits) {
|
|
223
|
+
const family = net.isIP(ip);
|
|
224
|
+
if (family === 4) return ipv4InCidr(ip, base, bits);
|
|
225
|
+
if (family === 6) return ipv6InCidr(ip, base, bits);
|
|
226
|
+
return false;
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
function bypassesProxy(target, noProxy) {
|
|
230
|
+
const list = noProxy.trim();
|
|
231
|
+
if (!list) return false;
|
|
232
|
+
const hostname = target.hostname.toLowerCase().replace(/^\[|\]$/g, '');
|
|
233
|
+
const port = target.port || (target.protocol === 'https:' ? '443' : '80');
|
|
234
|
+
for (let entry of list.split(',')) {
|
|
235
|
+
entry = entry.trim();
|
|
236
|
+
if (!entry) continue;
|
|
237
|
+
if (entry === '*') return true;
|
|
238
|
+
const { host: rawHost, port: entryPort } = splitHostPort(entry);
|
|
239
|
+
if (entryPort && entryPort !== port) continue;
|
|
240
|
+
let pattern = rawHost.toLowerCase().replace(/^\[|\]$/g, '');
|
|
241
|
+
const wildcard = pattern.startsWith('*.');
|
|
242
|
+
if (wildcard) pattern = pattern.slice(2);
|
|
243
|
+
const slash = pattern.indexOf('/');
|
|
244
|
+
if (slash !== -1 && net.isIP(pattern.slice(0, slash))) {
|
|
245
|
+
const bits = Number(pattern.slice(slash + 1));
|
|
246
|
+
if (Number.isInteger(bits) && net.isIP(hostname) && ipInCidr(hostname, pattern.slice(0, slash), bits)) {
|
|
247
|
+
return true;
|
|
248
|
+
}
|
|
249
|
+
continue;
|
|
250
|
+
}
|
|
251
|
+
const host = pattern.replace(/^\./, '');
|
|
252
|
+
if (!host) continue;
|
|
253
|
+
if (wildcard) {
|
|
254
|
+
if (hostname.endsWith(`.${host}`)) return true;
|
|
255
|
+
continue;
|
|
256
|
+
}
|
|
257
|
+
if (hostname === host || hostname.endsWith(`.${host}`)) return true;
|
|
258
|
+
}
|
|
259
|
+
return false;
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
/**
|
|
263
|
+
* Pick a proxy for this URL from HTTP_PROXY / HTTPS_PROXY / NO_PROXY (and
|
|
264
|
+
* their lowercase forms). The proxy host is not run through assertSafeTarget:
|
|
265
|
+
* corporate proxies live on loopback or RFC 1918 addresses, and they are not
|
|
266
|
+
* the page being fetched. Redirect hops still go through that check.
|
|
267
|
+
* @param {string} url
|
|
268
|
+
* @param {NodeJS.ProcessEnv} [env]
|
|
269
|
+
* @returns {string | null} proxy URL, or null to connect directly
|
|
270
|
+
*/
|
|
271
|
+
export function resolveProxy(url, env = process.env) {
|
|
272
|
+
const target = new URL(url);
|
|
273
|
+
if (bypassesProxy(target, envFirst(env, 'NO_PROXY', 'no_proxy') ?? '')) return null;
|
|
274
|
+
const httpsProxy = envFirst(env, 'HTTPS_PROXY', 'https_proxy');
|
|
275
|
+
const httpProxy = envFirst(env, 'HTTP_PROXY', 'http_proxy');
|
|
276
|
+
const chosen = target.protocol === 'https:' ? (httpsProxy || httpProxy) : httpProxy;
|
|
277
|
+
const normalized = normalizeProxy(chosen);
|
|
278
|
+
if (!normalized) return null;
|
|
279
|
+
assertHttpProxyProtocol(new URL(normalized));
|
|
280
|
+
return normalized;
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
function decodeCredential(value) {
|
|
284
|
+
try {
|
|
285
|
+
return decodeURIComponent(value);
|
|
286
|
+
} catch {
|
|
287
|
+
return value;
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
|
|
291
|
+
function proxyAuthHeader(proxy) {
|
|
292
|
+
if (!proxy.username) return undefined;
|
|
293
|
+
const token = Buffer.from(`${decodeCredential(proxy.username)}:${decodeCredential(proxy.password)}`).toString('base64');
|
|
294
|
+
return `Basic ${token}`;
|
|
295
|
+
}
|
|
296
|
+
|
|
297
|
+
function authority(target) {
|
|
298
|
+
const host = net.isIP(target.hostname) === 6 ? `[${target.hostname}]` : target.hostname;
|
|
299
|
+
const port = target.port || (target.protocol === 'https:' ? '443' : '80');
|
|
300
|
+
return `${host}:${port}`;
|
|
301
|
+
}
|
|
302
|
+
|
|
303
|
+
function wrapNodeResponse(res, url) {
|
|
304
|
+
const headers = {
|
|
305
|
+
get(name) {
|
|
306
|
+
const v = res.headers[name.toLowerCase()];
|
|
307
|
+
if (v == null) return null;
|
|
308
|
+
return Array.isArray(v) ? v.join(', ') : v;
|
|
309
|
+
},
|
|
310
|
+
};
|
|
311
|
+
const text = () => new Promise((resolve, reject) => {
|
|
312
|
+
const chunks = [];
|
|
313
|
+
res.on('data', (c) => chunks.push(c));
|
|
314
|
+
res.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
|
|
315
|
+
res.on('error', reject);
|
|
316
|
+
});
|
|
317
|
+
const status = res.statusCode ?? 0;
|
|
318
|
+
return {
|
|
319
|
+
status,
|
|
320
|
+
statusText: res.statusMessage || '',
|
|
321
|
+
headers,
|
|
322
|
+
ok: status >= 200 && status < 300,
|
|
323
|
+
url,
|
|
324
|
+
text,
|
|
325
|
+
};
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
function proxyTransport(proxy) {
|
|
329
|
+
return proxy.protocol === 'https:' ? https : http;
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
function proxyPort(proxy) {
|
|
333
|
+
return Number(proxy.port) || (proxy.protocol === 'https:' ? 443 : 80);
|
|
334
|
+
}
|
|
335
|
+
|
|
336
|
+
function armRequestTimeout(req, reject, label) {
|
|
337
|
+
req.setTimeout(PROXY_TIMEOUT_MS, () => {
|
|
338
|
+
req.destroy();
|
|
339
|
+
reject(new Error(`proxy timed out after ${PROXY_TIMEOUT_MS / 1000}s for ${label}`));
|
|
340
|
+
});
|
|
341
|
+
}
|
|
342
|
+
|
|
343
|
+
function pickTlsCa(tlsOpts) {
|
|
344
|
+
return tlsOpts.ca != null ? { ca: tlsOpts.ca } : {};
|
|
345
|
+
}
|
|
346
|
+
|
|
347
|
+
function httpViaProxy(target, proxy, headers) {
|
|
348
|
+
const auth = proxyAuthHeader(proxy);
|
|
349
|
+
return new Promise((resolve, reject) => {
|
|
350
|
+
const req = proxyTransport(proxy).request({
|
|
351
|
+
hostname: proxy.hostname,
|
|
352
|
+
port: proxyPort(proxy),
|
|
353
|
+
method: 'GET',
|
|
354
|
+
path: `${target.protocol}//${target.host}${target.pathname}${target.search}`,
|
|
355
|
+
headers: {
|
|
356
|
+
...headers,
|
|
357
|
+
host: target.host,
|
|
358
|
+
...(auth && { 'proxy-authorization': auth }),
|
|
359
|
+
},
|
|
360
|
+
}, (res) => resolve(wrapNodeResponse(res, target.href)));
|
|
361
|
+
armRequestTimeout(req, reject, target.href);
|
|
362
|
+
req.on('error', (err) => reject(new Error(`proxy failed: ${err.message} for ${target.href}`)));
|
|
363
|
+
req.end();
|
|
364
|
+
});
|
|
365
|
+
}
|
|
366
|
+
|
|
367
|
+
function httpsViaConnect(target, proxy, headers, tlsOpts = {}) {
|
|
368
|
+
const dest = authority(target);
|
|
369
|
+
const auth = proxyAuthHeader(proxy);
|
|
370
|
+
return new Promise((resolve, reject) => {
|
|
371
|
+
let settled = false;
|
|
372
|
+
const fail = (err) => {
|
|
373
|
+
if (settled) return;
|
|
374
|
+
settled = true;
|
|
375
|
+
req.destroy();
|
|
376
|
+
reject(err instanceof Error ? err : new Error(String(err)));
|
|
377
|
+
};
|
|
378
|
+
const req = proxyTransport(proxy).request({
|
|
379
|
+
hostname: proxy.hostname,
|
|
380
|
+
port: proxyPort(proxy),
|
|
381
|
+
method: 'CONNECT',
|
|
382
|
+
path: dest,
|
|
383
|
+
headers: {
|
|
384
|
+
host: dest,
|
|
385
|
+
...(auth && { 'proxy-authorization': auth }),
|
|
386
|
+
},
|
|
387
|
+
});
|
|
388
|
+
armRequestTimeout(req, fail, target.href);
|
|
389
|
+
req.on('connect', (res, socket, head) => {
|
|
390
|
+
if (res.statusCode !== 200) {
|
|
391
|
+
socket.destroy();
|
|
392
|
+
fail(new Error(`proxy CONNECT failed: ${res.statusCode} for ${target.href}`));
|
|
393
|
+
return;
|
|
394
|
+
}
|
|
395
|
+
if (head.length) socket.unshift(head);
|
|
396
|
+
// tls.connect already opened the tunnel. https.request would wrap TLS
|
|
397
|
+
// again, and the origin would see a second ClientHello as garbage.
|
|
398
|
+
// SNI is a hostname; an IP literal is only used for the cert check.
|
|
399
|
+
const hostname = target.hostname;
|
|
400
|
+
const tlsSocket = tls.connect({
|
|
401
|
+
socket,
|
|
402
|
+
host: hostname,
|
|
403
|
+
...(net.isIP(hostname) ? {} : { servername: hostname }),
|
|
404
|
+
...pickTlsCa(tlsOpts),
|
|
405
|
+
}, () => {
|
|
406
|
+
const tunneled = http.request({
|
|
407
|
+
createConnection: () => tlsSocket,
|
|
408
|
+
path: `${target.pathname}${target.search}`,
|
|
409
|
+
method: 'GET',
|
|
410
|
+
headers: { ...headers, host: target.host },
|
|
411
|
+
}, (httpsRes) => {
|
|
412
|
+
if (settled) return;
|
|
413
|
+
settled = true;
|
|
414
|
+
resolve(wrapNodeResponse(httpsRes, target.href));
|
|
415
|
+
});
|
|
416
|
+
armRequestTimeout(tunneled, fail, target.href);
|
|
417
|
+
tunneled.on('error', (err) => fail(new Error(`proxy failed: ${err.message || err.code} for ${target.href}`)));
|
|
418
|
+
tunneled.end();
|
|
419
|
+
});
|
|
420
|
+
tlsSocket.on('error', (err) => fail(new Error(`proxy failed: ${err.message || err.code} for ${target.href}`)));
|
|
421
|
+
});
|
|
422
|
+
req.on('error', (err) => fail(new Error(`proxy failed: ${err.message || err.code} for ${target.href}`)));
|
|
423
|
+
req.end();
|
|
424
|
+
});
|
|
425
|
+
}
|
|
426
|
+
|
|
427
|
+
/**
|
|
428
|
+
* One GET through an HTTP(S) proxy. HTTP targets use the absolute-URI form;
|
|
429
|
+
* HTTPS targets open a CONNECT tunnel first. Redirects are not followed:
|
|
430
|
+
* followRedirects owns that so each hop still goes through assertSafeTarget.
|
|
431
|
+
* @param {string} url
|
|
432
|
+
* @param {string} proxy
|
|
433
|
+
* @param {Record<string, string>} [headers]
|
|
434
|
+
* @param {import('node:tls').ConnectionOptions} [tlsOpts]
|
|
435
|
+
*/
|
|
436
|
+
export function proxyGet(url, proxy, headers = {}, tlsOpts = {}) {
|
|
437
|
+
const target = new URL(url);
|
|
438
|
+
const proxyUrl = new URL(proxy);
|
|
439
|
+
assertHttpProxyProtocol(proxyUrl);
|
|
440
|
+
return target.protocol === 'https:'
|
|
441
|
+
? httpsViaConnect(target, proxyUrl, headers, tlsOpts)
|
|
442
|
+
: httpViaProxy(target, proxyUrl, headers);
|
|
443
|
+
}
|
|
444
|
+
|
|
149
445
|
/**
|
|
150
446
|
* Fetch a page.
|
|
151
447
|
* @param {string} url - with or without a scheme, https is assumed
|
|
@@ -160,11 +456,25 @@ export async function fetchPage(url) {
|
|
|
160
456
|
return impers ? viaImpers(impers, target) : viaFetch(target);
|
|
161
457
|
}
|
|
162
458
|
|
|
163
|
-
|
|
164
|
-
|
|
459
|
+
/**
|
|
460
|
+
* Follow redirects one hop at a time, validating each destination before the
|
|
461
|
+
* next request goes out.
|
|
462
|
+
*
|
|
463
|
+
* Both transports share this loop. They used to carry one each, which made the
|
|
464
|
+
* check that matters something a change could fix in one place and leave broken
|
|
465
|
+
* in the other, and made the guarantee testable only through a third party
|
|
466
|
+
* willing to 302 wherever it was told. Taking the request as a callback is what
|
|
467
|
+
* lets the hop check be proven against a transport that never leaves the
|
|
468
|
+
* process.
|
|
469
|
+
* @param {(url: string) => Promise<any>} get - one request, redirects not followed
|
|
470
|
+
* @param {string} start
|
|
471
|
+
* @returns {Promise<{res: any, url: string}>} the first non-redirect response
|
|
472
|
+
*/
|
|
473
|
+
export async function followRedirects(get, start) {
|
|
474
|
+
let current = start;
|
|
165
475
|
for (let i = 0; ; i++) {
|
|
166
|
-
if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${
|
|
167
|
-
const res = await
|
|
476
|
+
if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${start}`);
|
|
477
|
+
const res = await get(current);
|
|
168
478
|
const status = res.status ?? res.statusCode ?? 0;
|
|
169
479
|
const location = res.headers.get('location');
|
|
170
480
|
if (status >= 300 && status < 400 && location) {
|
|
@@ -172,19 +482,27 @@ async function followImpersRedirects(impers, startUrl, impersonate) {
|
|
|
172
482
|
await assertSafeTarget(current);
|
|
173
483
|
continue;
|
|
174
484
|
}
|
|
175
|
-
return res;
|
|
485
|
+
return { res, url: current };
|
|
176
486
|
}
|
|
177
487
|
}
|
|
178
488
|
|
|
489
|
+
const FETCH_HEADERS = {
|
|
490
|
+
'user-agent': UA,
|
|
491
|
+
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
|
492
|
+
'accept-language': 'en-US,en;q=0.9',
|
|
493
|
+
};
|
|
494
|
+
|
|
179
495
|
async function viaImpers(impers, target) {
|
|
180
496
|
// Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a
|
|
181
497
|
// blocked first attempt gets one cheap retry with a second identity.
|
|
498
|
+
const asking = (impersonate) => (url) =>
|
|
499
|
+
impers.get(url, { impersonate, allowRedirects: false, proxy: resolveProxy(url) ?? '' });
|
|
182
500
|
let via = 'impers:chrome';
|
|
183
|
-
let res = await
|
|
501
|
+
let { res } = await followRedirects(asking('chrome'), target);
|
|
184
502
|
let status = res.status ?? res.statusCode ?? 0;
|
|
185
503
|
if (status >= 400) {
|
|
186
504
|
via = 'impers:firefox';
|
|
187
|
-
res = await
|
|
505
|
+
({ res } = await followRedirects(asking('firefox'), target));
|
|
188
506
|
status = res.status ?? res.statusCode ?? 0;
|
|
189
507
|
}
|
|
190
508
|
if (status >= 400) throw new Error(`fetch failed: ${status} for ${target}`);
|
|
@@ -194,26 +512,12 @@ async function viaImpers(impers, target) {
|
|
|
194
512
|
}
|
|
195
513
|
|
|
196
514
|
async function viaFetch(target) {
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
headers: {
|
|
204
|
-
'user-agent': UA,
|
|
205
|
-
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
|
206
|
-
'accept-language': 'en-US,en;q=0.9',
|
|
207
|
-
},
|
|
208
|
-
});
|
|
209
|
-
const location = res.headers.get('location');
|
|
210
|
-
if (res.status >= 300 && res.status < 400 && location) {
|
|
211
|
-
current = new URL(location, current).toString();
|
|
212
|
-
await assertSafeTarget(current);
|
|
213
|
-
continue;
|
|
214
|
-
}
|
|
215
|
-
break;
|
|
216
|
-
}
|
|
515
|
+
const { res, url: current } = await followRedirects((url) => {
|
|
516
|
+
const proxy = resolveProxy(url);
|
|
517
|
+
return proxy
|
|
518
|
+
? proxyGet(url, proxy, FETCH_HEADERS)
|
|
519
|
+
: fetch(url, { redirect: 'manual', headers: FETCH_HEADERS });
|
|
520
|
+
}, target);
|
|
217
521
|
if (!res.ok) {
|
|
218
522
|
throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${current}`);
|
|
219
523
|
}
|
package/src/render.js
CHANGED
|
@@ -16,6 +16,62 @@ export const estimateTokens = (s) => Math.ceil(s.length / 4);
|
|
|
16
16
|
|
|
17
17
|
const num = (v) => v.toLocaleString('en-US');
|
|
18
18
|
|
|
19
|
+
// A page that distilled to nothing must not read like a page with nothing on
|
|
20
|
+
// it. JS-only pages, consent walls, and bot challenges all answer HTTP 200 with
|
|
21
|
+
// markup carrying no text, and "100% saved" is technically true of a render
|
|
22
|
+
// that saved every token by extracting none. From the output alone an agent
|
|
23
|
+
// cannot tell that from a genuinely empty page, so it never falls back to
|
|
24
|
+
// something heavier and an empty result travels on as evidence. oc has to fail
|
|
25
|
+
// loud and cheap there instead, per principle 5 in CONTRIBUTING.
|
|
26
|
+
//
|
|
27
|
+
// The numbers below are measured, not guessed. Against live pages, the thinnest
|
|
28
|
+
// render the README claims (an X profile) carries about 460 tokens of content,
|
|
29
|
+
// while a JS-only or gated one carries under 55 (reddit.com/r/*,
|
|
30
|
+
// instagram.com), so a threshold in between never has to be a close call.
|
|
31
|
+
|
|
32
|
+
// Text the page itself wrote, in tokens: prose and headings, plus link and
|
|
33
|
+
// button labels long enough to be content rather than furniture. Nav chrome is
|
|
34
|
+
// short by nature ("Help", "Log in") and a headline or a post title is not,
|
|
35
|
+
// which is what tells a link-list page (Hacker News, search results) from a
|
|
36
|
+
// page whose only links are its own menu.
|
|
37
|
+
const CONTENT_LABEL = 25;
|
|
38
|
+
// Below this there is nothing to read whatever the page is, so how much markup
|
|
39
|
+
// it arrived in does not matter.
|
|
40
|
+
export const MIN_CONTENT = 25;
|
|
41
|
+
// Below this, with markup that large behind it, the fetch worked and the render
|
|
42
|
+
// did not: a real page of that weight always distills to more. A genuinely
|
|
43
|
+
// short page is exempt, because its HTML never reaches THIN_HTML.
|
|
44
|
+
const THIN_CONTENT = 100;
|
|
45
|
+
const THIN_HTML = 2500;
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* How much of a distilled page is text the page itself wrote.
|
|
49
|
+
* @param {import('./distill.js').Page} page
|
|
50
|
+
* @returns {number}
|
|
51
|
+
*/
|
|
52
|
+
export const contentTokens = (page) =>
|
|
53
|
+
page.blocks.reduce((sum, b) => {
|
|
54
|
+
if (b.type === 'heading' || b.type === 'text') return sum + estimateTokens(b.text ?? '');
|
|
55
|
+
const label = (b.type === 'link' || b.type === 'button') && (b.text ?? '').length > CONTENT_LABEL;
|
|
56
|
+
return label ? sum + estimateTokens(b.text) : sum;
|
|
57
|
+
}, 0);
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* Why this render carries no content worth printing, as a phrase for the
|
|
61
|
+
* failure line, or null when it does carry some. Both figures are token counts,
|
|
62
|
+
* which is the unit the caller is paying in.
|
|
63
|
+
* @param {number} content
|
|
64
|
+
* @param {number} htmlTokens
|
|
65
|
+
* @returns {string|null}
|
|
66
|
+
*/
|
|
67
|
+
export function contentFailure(content, htmlTokens) {
|
|
68
|
+
if (content < MIN_CONTENT) return `~${content} tokens of text on the whole page`;
|
|
69
|
+
if (content < THIN_CONTENT && htmlTokens > THIN_HTML) {
|
|
70
|
+
return `~${content} tokens of text out of ~${htmlTokens} of HTML`;
|
|
71
|
+
}
|
|
72
|
+
return null;
|
|
73
|
+
}
|
|
74
|
+
|
|
19
75
|
// How far past the budget a page may run and still be printed whole. Cutting a
|
|
20
76
|
// page that was nearly done costs the agent a second command, and a command is
|
|
21
77
|
// dear: measured inside Claude Code, one tool call is 23,000 to 33,000 tokens of
|
|
@@ -174,6 +230,12 @@ const SENTENCE_FLOOR = 0.7;
|
|
|
174
230
|
|
|
175
231
|
const truncate = (s) => {
|
|
176
232
|
if (s.length <= TEXT_CAP) return s;
|
|
233
|
+
// A line is to code what a sentence is to prose, and a code block is the only
|
|
234
|
+
// text that keeps its newlines, so the same courtesy applies: cut where a
|
|
235
|
+
// line ended. A period in code ends nothing, which is why this returns
|
|
236
|
+
// instead of falling through to the sentence rule below.
|
|
237
|
+
const line = s.slice(0, TEXT_CAP).lastIndexOf('\n');
|
|
238
|
+
if (line >= TEXT_CAP * SENTENCE_FLOOR) return `${s.slice(0, line)} ... +${num(s.length - line)} chars`;
|
|
177
239
|
let cut = TEXT_CAP;
|
|
178
240
|
for (const m of s.slice(0, TEXT_CAP).matchAll(SENTENCE_END)) {
|
|
179
241
|
const end = (m.index ?? 0) + m[0].length;
|
package/src/sites.js
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Site shortcuts. `clis/*.json` names the URLs on a site worth reaching
|
|
3
|
+
* directly, so `oc hn item 4711` gets there without the agent knowing that
|
|
4
|
+
* Hacker News spells it /item?id=. A shortcut is only ever a URL: it resolves
|
|
5
|
+
* to one and hands off to the same fetch and render path `oc open` uses, so
|
|
6
|
+
* nothing here can change what a page costs or how it reads.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
import { readdirSync, readFileSync } from 'node:fs';
|
|
10
|
+
import { fileURLToPath } from 'node:url';
|
|
11
|
+
|
|
12
|
+
const DIR = fileURLToPath(new URL('../clis/', import.meta.url));
|
|
13
|
+
|
|
14
|
+
// Short names an agent is likely to reach for. The domain itself and its
|
|
15
|
+
// registrable label always resolve, so this only covers what those miss.
|
|
16
|
+
const ALIASES = {
|
|
17
|
+
hn: 'news.ycombinator.com',
|
|
18
|
+
gh: 'github.com',
|
|
19
|
+
so: 'stackoverflow.com',
|
|
20
|
+
ddg: 'duckduckgo.com',
|
|
21
|
+
yt: 'youtube.com',
|
|
22
|
+
finance: 'finance.yahoo.com',
|
|
23
|
+
twitter: 'x.com',
|
|
24
|
+
aws: 'docs.aws.amazon.com',
|
|
25
|
+
gcp: 'cloud.google.com',
|
|
26
|
+
learn: 'learn.microsoft.com',
|
|
27
|
+
wiki: 'wikipedia.org',
|
|
28
|
+
};
|
|
29
|
+
|
|
30
|
+
/** @typedef {{open: string, args?: string[]}} Shortcut */
|
|
31
|
+
/** @typedef {{domain: string, commands: Record<string, Shortcut>}} Site */
|
|
32
|
+
|
|
33
|
+
/** @type {Map<string, Site>|null} */
|
|
34
|
+
let cache = null;
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* Every site definition that ships with oc, keyed by each name that resolves
|
|
38
|
+
* to it. A definition that will not parse is skipped rather than fatal: a bad
|
|
39
|
+
* file costs its own shortcuts and leaves every other command working.
|
|
40
|
+
* @returns {Map<string, Site>}
|
|
41
|
+
*/
|
|
42
|
+
export function sites() {
|
|
43
|
+
if (cache) return cache;
|
|
44
|
+
cache = new Map();
|
|
45
|
+
/** @type {string[]} */
|
|
46
|
+
let files = [];
|
|
47
|
+
try {
|
|
48
|
+
files = readdirSync(DIR).filter((f) => f.endsWith('.json')).sort();
|
|
49
|
+
} catch {
|
|
50
|
+
return cache;
|
|
51
|
+
}
|
|
52
|
+
for (const file of files) {
|
|
53
|
+
/** @type {Site} */
|
|
54
|
+
let site;
|
|
55
|
+
try {
|
|
56
|
+
site = JSON.parse(readFileSync(DIR + file, 'utf8'));
|
|
57
|
+
} catch {
|
|
58
|
+
continue;
|
|
59
|
+
}
|
|
60
|
+
if (!site?.domain || !site.commands) continue;
|
|
61
|
+
const labels = site.domain.split('.');
|
|
62
|
+
for (const key of [site.domain, labels[labels.length - 2]]) {
|
|
63
|
+
// First definition wins, in sorted filename order, so which site owns a
|
|
64
|
+
// shared label never depends on how the directory happens to be read.
|
|
65
|
+
if (key && !cache.has(key)) cache.set(key, site);
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
for (const [alias, domain] of Object.entries(ALIASES)) {
|
|
69
|
+
const site = cache.get(domain);
|
|
70
|
+
if (site && !cache.has(alias)) cache.set(alias, site);
|
|
71
|
+
}
|
|
72
|
+
return cache;
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
const verbs = (site) =>
|
|
76
|
+
Object.entries(site.commands)
|
|
77
|
+
.map(([name, def]) => (def.args?.length ? `${name} <${def.args.join('> <')}>` : name))
|
|
78
|
+
.join(' | ');
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* Resolve `oc <site> <verb> [args]` to a URL, or null when the first word
|
|
82
|
+
* names no site oc ships, which is the caller's cue to report an unknown
|
|
83
|
+
* command. A site that exists with a verb that does not is an error here
|
|
84
|
+
* instead, since the agent has the right site and only needs the verb list.
|
|
85
|
+
* @param {string} name
|
|
86
|
+
* @param {string[]} args
|
|
87
|
+
* @returns {{url: string, domain: string, command: string}|null}
|
|
88
|
+
*/
|
|
89
|
+
export function resolveSite(name, args) {
|
|
90
|
+
const site = sites().get(name.toLowerCase());
|
|
91
|
+
if (!site) return null;
|
|
92
|
+
const [verb, ...rest] = args;
|
|
93
|
+
if (!verb) throw new Error(`usage: oc ${name} <verb>, one of: ${verbs(site)}`);
|
|
94
|
+
const def = site.commands[verb];
|
|
95
|
+
if (!def) throw new Error(`'${verb}' is not a ${site.domain} shortcut, try: ${verbs(site)}`);
|
|
96
|
+
|
|
97
|
+
const need = def.args ?? [];
|
|
98
|
+
if (rest.length < need.length) {
|
|
99
|
+
throw new Error(`usage: oc ${name} ${verb} <${need.join('> <')}>`);
|
|
100
|
+
}
|
|
101
|
+
// The last argument soaks up everything left, so a query an agent typed as
|
|
102
|
+
// separate words ('oc ddg search claude code cli') works unquoted.
|
|
103
|
+
const values = need.map((_, i) =>
|
|
104
|
+
i === need.length - 1 ? rest.slice(i).join(' ') : rest[i]);
|
|
105
|
+
const url = need.reduce(
|
|
106
|
+
(open, arg, i) => open.replaceAll(`{${arg}}`, encode(values[i], def.open, arg)),
|
|
107
|
+
def.open);
|
|
108
|
+
return { url, domain: site.domain, command: verb };
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
/**
|
|
112
|
+
* Percent-encode one value for the slot it fills. A value in the query string
|
|
113
|
+
* is encoded outright, but a docs path is often several segments deep, so
|
|
114
|
+
* 'oc learn doc azure/aks/intro' has to keep its slashes: escaping them would
|
|
115
|
+
* ask the site for one impossible segment instead of the page.
|
|
116
|
+
* @param {string} value
|
|
117
|
+
* @param {string} template
|
|
118
|
+
* @param {string} arg
|
|
119
|
+
* @returns {string}
|
|
120
|
+
*/
|
|
121
|
+
function encode(value, template, arg) {
|
|
122
|
+
const query = template.includes('?') && template.indexOf(`{${arg}}`) > template.indexOf('?');
|
|
123
|
+
const encoded = encodeURIComponent(value);
|
|
124
|
+
return query ? encoded : encoded.replaceAll('%2F', '/');
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/**
|
|
128
|
+
* One line per site for `oc sites`, shortest name first, since that is what an
|
|
129
|
+
* agent will type. Discovery has to cost less than a wrong guess does, so this
|
|
130
|
+
* stays one line each rather than a table.
|
|
131
|
+
* @returns {string}
|
|
132
|
+
*/
|
|
133
|
+
export function listSites() {
|
|
134
|
+
const byDomain = new Map();
|
|
135
|
+
for (const [key, site] of sites()) {
|
|
136
|
+
if (!byDomain.has(site.domain)) byDomain.set(site.domain, { site, keys: [] });
|
|
137
|
+
byDomain.get(site.domain).keys.push(key);
|
|
138
|
+
}
|
|
139
|
+
const lines = [...byDomain.values()].map(({ site, keys }) => {
|
|
140
|
+
const names = keys
|
|
141
|
+
.filter((k) => k !== site.domain)
|
|
142
|
+
.sort((a, b) => a.length - b.length || a.localeCompare(b));
|
|
143
|
+
return `oc ${names[0] ?? site.domain} <verb> (${[...names.slice(1), site.domain].join(', ')}): ${verbs(site)}`;
|
|
144
|
+
});
|
|
145
|
+
return lines.length
|
|
146
|
+
? `${lines.join('\n')}\nany other site: oc open <url>`
|
|
147
|
+
: 'no site definitions found; oc open <url> works on any site';
|
|
148
|
+
}
|