@only-cli/oc 0.5.2 → 0.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -11
- package/package.json +1 -1
- package/skills/web-browsing-cli/SKILL.md +24 -9
- package/src/cli.js +43 -8
- package/src/cookies.js +13 -7
- package/src/distill.js +9 -3
- package/src/fetch.js +68 -7
- package/src/session.js +63 -6
package/README.md
CHANGED
|
@@ -70,6 +70,7 @@ oc fill <n> <text> type into a numbered input (planned)
|
|
|
70
70
|
oc submit [n] submit a form (planned)
|
|
71
71
|
oc login seed cookies for a session (--cookie, --domain)
|
|
72
72
|
oc logout [session] forget a session: cookies and saved page
|
|
73
|
+
oc session ls|rm [name] list saved sessions, or forget one (page + cookies)
|
|
73
74
|
```
|
|
74
75
|
|
|
75
76
|
Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTML), `--session <name>`, `--verbose`/`-v` (metrics on stderr, or export `OC_VERBOSE=1`).
|
|
@@ -107,7 +108,7 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum
|
|
|
107
108
|
|
|
108
109
|
A shortcut only ever resolves to a URL and then takes the same path `oc open` does, so it changes nothing about what a page costs or how it reads. The last argument takes every word after it, so `oc ddg search claude code cli` and `oc aws search s3 lifecycle rules` need no quoting, and a path argument keeps its slashes, so `oc learn doc azure/aks/what-is-aks` reaches that page.
|
|
109
110
|
|
|
110
|
-
A few of these (X, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies).
|
|
111
|
+
A few of these (X, Reddit, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. Reddit goes through the Atom feeds on www.reddit.com: old.reddit.com has sent logged-out readers to a login page since June 2026 and the `.json` views answer 403 without an OAuth token, so `oc open` on a reddit.com front page, subreddit, post, user or search URL reads the matching feed instead when no login session is held, which is also how `oc do <n>` on a post in a subreddit feed reaches its comments; `oc reddit post <id>` is the same feed by hand. oc asks reddit.com with its Firefox fingerprint first, because Reddit's edge refuses the Chrome one more often than not. The feeds carry titles, authors, dates, and bodies but no scores or comment counts, and anonymous reddit.com meters each client tightly: a second request within half a minute of the first has come back 429 in testing, and a burst of Reddit shortcuts ends in refusals that take minutes to clear, so space them out. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies).
|
|
111
112
|
|
|
112
113
|
Want a website on that list? Open a pull request, or an issue naming the site; see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
113
114
|
|
|
@@ -123,6 +124,15 @@ oc open https://example.com/dashboard --session work
|
|
|
123
124
|
oc logout work
|
|
124
125
|
```
|
|
125
126
|
|
|
127
|
+
Windows (PowerShell, no `printf` there):
|
|
128
|
+
|
|
129
|
+
```powershell
|
|
130
|
+
$h = 'session=...; auth=...' # paste from browser devtools
|
|
131
|
+
$h | oc login --cookie - --domain example.com --expires 2h --session work
|
|
132
|
+
oc open https://example.com/dashboard --session work
|
|
133
|
+
oc logout work
|
|
134
|
+
```
|
|
135
|
+
|
|
126
136
|
Prefer `--cookie -`, which reads the header from stdin. The flag also takes the header inline (`--cookie "session=..."`), but an argument is a live credential in `ps` for as long as `oc` runs and in your shell history afterwards.
|
|
127
137
|
|
|
128
138
|
Copy the `Cookie` header from your browser's devtools (Application → Cookies, or the Network tab on a request); a leading `Cookie:` is stripped for you. `--domain` is the site hostname those cookies belong to, and it has to be a real hostname: a bare TLD like `com` is refused, because the match is a suffix match and those cookies would go to every `.com` host the session ever fetched. Cookie names and values are checked at login too, so a stray control character fails there rather than deep inside the HTTP client.
|
|
@@ -131,7 +141,7 @@ Seeded cookies are https-only. They almost always come from an https browser ses
|
|
|
131
141
|
|
|
132
142
|
Cookies live in a separate sidecar file (`<session>.cookies.json`) under `~/.only-cli/sessions/`, mode `0600`, not in the page-state JSON and never in `--json` output. The default lifetime is one hour (`--expires 1h`), and a jar holds at most 50 cookies so a page cannot bloat it. When cookies expire or the site returns a login page, `oc` says so plainly (exit 2) instead of distilling the login form as content.
|
|
133
143
|
|
|
134
|
-
`oc logout` forgets the whole session, not just its cookies: a page saved under that name can hold the distilled text of something only the login could reach, so the snapshot goes with the jar.
|
|
144
|
+
`oc logout` forgets the whole session, not just its cookies: a page saved under that name can hold the distilled text of something only the login could reach, so the snapshot goes with the jar. `oc session rm [name]` forgets a session the same way (page plus cookies) without switching to it first, and `oc session ls` lists what is on disk (`[cookies]` marks a live jar). State lives in `~/.only-cli` (`%USERPROFILE%\.only-cli` on Windows, override with `OC_HOME`), one JSON per session plus search-index caches. Delete the directory to start over.
|
|
135
145
|
|
|
136
146
|
`oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. A result title on a search page is a link, so `oc do` on it opens the result rather than repeating the title. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page, and a `find` with a single match prints that region instead of the number to read it with. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
|
|
137
147
|
|
|
@@ -168,26 +178,26 @@ Private and internal addresses are refused whether or not a proxy is set. With a
|
|
|
168
178
|
|
|
169
179
|
## Benchmarks
|
|
170
180
|
|
|
171
|
-
Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.1, September 2026, live sites):
|
|
181
|
+
Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.1 to 0.5.3, September 2026, live sites):
|
|
172
182
|
|
|
173
|
-
- **
|
|
174
|
-
- **Real content on
|
|
183
|
+
- **118x fewer tokens than raw HTML** across the fourteen real pages both could read: 9,466 against 1,119,003. 17x fewer than Jina Reader, 54x fewer than Playwright MCP's accessibility snapshot.
|
|
184
|
+
- **Real content on all fifteen pages, Reddit included.** Reddit's pages now sit behind a login wall, so the suite reads its thread and subreddit through the Atom feeds that still answer anonymously, the same route the `oc reddit` shortcuts take: 459 and 493 tokens through oc against 10,452 and 21,155 for the feed XML, while the browser-based tools and Jina Reader got the block pages Reddit serves their fingerprints. Yahoo Finance refuses plain fetch outright and DuckDuckGo still blocks lynx; oc's Chrome impersonation read both.
|
|
175
185
|
- **Half the cost of Claude Code's built-in `WebSearch`** on Wikipedia lookups: $0.23 against $0.45 for five questions, both 5/5 correct, on 25x less fresh input.
|
|
176
186
|
- **21% cheaper than `WebFetch` and 34% cheaper than `WebSearch`** on eleven language docs lookups, at equal or better accuracy.
|
|
177
187
|
- **10% cheaper than `WebFetch` and 49% cheaper than `WebSearch`** on twelve dependency lookups across GitHub, npm, PyPI, RubyGems, crates.io, Docker Hub, Stack Overflow and an RFC, 12/12 correct with no tuned shortcut for most of those sites.
|
|
178
188
|
|
|
179
189
|
The tables behind those numbers:
|
|
180
190
|
|
|
181
|
-
**Tokens per page, no model in the loop.** Fifteen real pages: a news front page, a Reddit
|
|
191
|
+
**Tokens per page, no model in the loop.** Fifteen real pages: a news front page, a Reddit thread and subreddit through their Atom feeds, search results, a stock quote, three cloud CLI references, the Python, MDN, and Node.js references, and more.
|
|
182
192
|
|
|
183
193
|
| method | tokens for 15 pages | notes |
|
|
184
194
|
| --- | ---: | --- |
|
|
185
|
-
| `oc open` |
|
|
186
|
-
| Jina Reader |
|
|
187
|
-
| Playwright MCP |
|
|
188
|
-
| raw HTML fetch | 1,
|
|
195
|
+
| `oc open` | 9,913 | real content on all 15 pages; the two Reddit feeds cost 459 and 493 tokens |
|
|
196
|
+
| Jina Reader | 145,679 | both Reddit results are block pages; failed LinkedIn outright |
|
|
197
|
+
| Playwright MCP | 535,908 | accessibility snapshots; both Reddit snapshots are block pages, since Reddit refuses the Chrome fingerprint |
|
|
198
|
+
| raw HTML fetch | 1,119,003 | the two Reddit feeds are 10,452 and 21,155 tokens of XML; Yahoo Finance refused the connection |
|
|
189
199
|
|
|
190
|
-
oc's budget keeps every page near 500 tokens however much it weighs: the YouTube watch page is
|
|
200
|
+
oc's budget keeps every page near 500 tokens however much it weighs: the YouTube watch page is 363,516 tokens raw and 688 through oc, Node's `fs` reference 275,425 against 479. On the fourteen pages both could read, raw HTML costs 118x what oc does: 1,119,003 against 9,466. Reddit meters anonymous feed readers, so the suite spaces its Reddit requests a minute apart; a second request from the same client inside thirty seconds came back 429 in every faster pass.
|
|
191
201
|
|
|
192
202
|
**Whole tasks against the agent's built-in web tools.** Read cost is one thing, what an agent actually spends is another, so a second set of suites runs full lookups end to end in Claude Code (`claude-sonnet-5`), one tool per run, and grades every answer. Five Wikipedia lookups, eleven language documentation lookups, and twelve lookups on the pages around a dependency, where oc has shortcuts only for GitHub and Stack Overflow and renders the rest generically:
|
|
193
203
|
|
package/package.json
CHANGED
|
@@ -8,15 +8,17 @@ description: Token-efficient web browsing and web content extraction for AI agen
|
|
|
8
8
|
Renders a web page as a compact, numbered terminal view instead of raw HTML. A typical page is under 500 tokens.
|
|
9
9
|
|
|
10
10
|
```
|
|
11
|
-
npx --yes @only-cli/oc@0.5.
|
|
12
|
-
npx --yes @only-cli/oc@0.5.
|
|
13
|
-
npx --yes @only-cli/oc@0.5.
|
|
11
|
+
npx --yes @only-cli/oc@0.5.4 open <url> compact view, numbered elements
|
|
12
|
+
npx --yes @only-cli/oc@0.5.4 do <n> follow link [n], or read it if [n] is text
|
|
13
|
+
npx --yes @only-cli/oc@0.5.4 find <query> where a string appears, or that place itself
|
|
14
14
|
when only one matches
|
|
15
|
-
npx --yes @only-cli/oc@0.5.
|
|
16
|
-
npx --yes @only-cli/oc@0.5.
|
|
17
|
-
npx --yes @only-cli/oc@0.5.
|
|
18
|
-
npx --yes @only-cli/oc@0.5.
|
|
19
|
-
npx --yes @only-cli/oc@0.5.
|
|
15
|
+
npx --yes @only-cli/oc@0.5.4 next next ~500 tokens of the page already open
|
|
16
|
+
npx --yes @only-cli/oc@0.5.4 read <n> full text of region [n]
|
|
17
|
+
npx --yes @only-cli/oc@0.5.4 raw [url] whole page as markdown (--html for cleaned HTML)
|
|
18
|
+
npx --yes @only-cli/oc@0.5.4 login seed cookies (--cookie, --domain, --expires)
|
|
19
|
+
npx --yes @only-cli/oc@0.5.4 logout [session] forget a session: cookies and saved page
|
|
20
|
+
npx --yes @only-cli/oc@0.5.4 session ls list saved sessions (name, url, title)
|
|
21
|
+
npx --yes @only-cli/oc@0.5.4 session rm [name] forget a saved session: page and cookies
|
|
20
22
|
```
|
|
21
23
|
|
|
22
24
|
None of these except `open`/`do`/`raw <url>` fetch anything; they replay the page `open` already saved.
|
|
@@ -32,7 +34,7 @@ oc ddg search claude code oc so question 231767 oc learn doc azur
|
|
|
32
34
|
oc py library json oc mdn js Array/map oc node api fs
|
|
33
35
|
```
|
|
34
36
|
|
|
35
|
-
Sites: `hn`, `reddit`, `gh`, `x`, `linkedin`, `ddg`, `bing`, `so`, `yahoo`, `yt`, `aws`, `gcp`, `learn`, `wiki`, `py`, `mdn`, `node`, `ruby`, `go`, `rust`, `java`, `php`, `cpp`, `ts`. Name one by short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`). The last argument takes every word after it, so a query or title needs no quoting. `oc sites` lists every site with its verbs, which is cheaper than guessing one.
|
|
37
|
+
Sites: `hn`, `reddit`, `gh`, `x`, `linkedin`, `ddg`, `bing`, `so`, `yahoo`, `yt`, `aws`, `gcp`, `learn`, `wiki`, `py`, `mdn`, `node`, `ruby`, `go`, `rust`, `java`, `php`, `cpp`, `ts`. Name one by short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`). The last argument takes every word after it, so a query or title needs no quoting. `oc sites` lists every site with its verbs, which is cheaper than guessing one. Reddit reads through its Atom feeds: a reddit.com page URL given to `open` meets a login wall or a 403, so use `oc reddit post <id>` or add `/.rss` to the URL, and keep Reddit calls under about ten a minute or the site answers 429.
|
|
36
38
|
|
|
37
39
|
Prefer a shortcut over a hand-built URL when one exists for the site, and prefer `oc wiki article <title>` over a search when you already know the article's name.
|
|
38
40
|
|
|
@@ -86,10 +88,23 @@ oc open https://example.com/dashboard --session work
|
|
|
86
88
|
oc logout work
|
|
87
89
|
```
|
|
88
90
|
|
|
91
|
+
Windows (PowerShell, no `printf`):
|
|
92
|
+
|
|
93
|
+
```powershell
|
|
94
|
+
$h = 'session=...; auth=...'
|
|
95
|
+
$h | oc login --cookie - --domain example.com --expires 2h --session work
|
|
96
|
+
oc open https://example.com/dashboard --session work
|
|
97
|
+
oc logout work
|
|
98
|
+
```
|
|
99
|
+
|
|
89
100
|
Pass `--cookie -` and pipe the header in, as above: an inline `--cookie "session=..."` puts a live credential in `ps` and in shell history. Copy the header from browser devtools. `--domain` must be a real hostname; a bare TLD like `com` is refused, since the cookies would then go to every `.com` host the session fetched.
|
|
90
101
|
|
|
91
102
|
Default lifetime is 1h. Seeded cookies are https-only: they are never sent over plain `http`, including on a redirect that downgrades, unless you seeded them with `--allow-http`. When cookies expire or the site returns a login page, `oc` says so (exit 2) instead of rendering the login form as content. Cookies live in a separate file from page state and are never included in `--json` output. `oc logout` drops that session's saved page along with its cookies.
|
|
92
103
|
|
|
104
|
+
## Saved sessions live on disk
|
|
105
|
+
|
|
106
|
+
State is one JSON per session under `~/.only-cli/sessions/` (`%USERPROFILE%\.only-cli\sessions\` on Windows, `OC_HOME` overrides). `session ls` shows what accumulated; `session rm [name]` drops a session's page and cookies (same promise as `logout`). Deleting the directory starts over.
|
|
107
|
+
|
|
93
108
|
## When not to use it
|
|
94
109
|
|
|
95
110
|
Pages needing heavy client-side JS aren't supported yet. A page with no readable text (JavaScript-only, a consent wall, a bot challenge) prints one line on stderr and exits 2, which is distinct from the exit 1 every other failure uses, so exit 2 means "oc cannot read this one" rather than "this page is empty". Take it at its word: say so and fall back to another tool rather than retrying the same URL.
|
package/src/cli.js
CHANGED
|
@@ -10,7 +10,7 @@ import { nodeSearch } from './nodedocs.js';
|
|
|
10
10
|
import { rdocSearch } from './rdoc.js';
|
|
11
11
|
import { apiSearch } from './apisearch.js';
|
|
12
12
|
import * as act from './act.js';
|
|
13
|
-
import { DEFAULT_SESSION, assertSafeName, clearSession, loadSession, saveSession, sessionFromPage } from './session.js';
|
|
13
|
+
import { DEFAULT_SESSION, assertSafeName, clearSession, listSessions, loadSession, saveSession, sessionFromPage } from './session.js';
|
|
14
14
|
import { authFailure, sessionExpiredMessage } from './auth.js';
|
|
15
15
|
import {
|
|
16
16
|
loadCookieJar,
|
|
@@ -42,7 +42,7 @@ usage: oc <command> [args] [flags]
|
|
|
42
42
|
back return to the previous page (planned)
|
|
43
43
|
login seed cookies for a session (--cookie, --domain)
|
|
44
44
|
logout [session] forget a session: its cookies and its saved page
|
|
45
|
-
session ls|rm
|
|
45
|
+
session ls|rm [name] list saved sessions, or forget one (page + cookies)
|
|
46
46
|
|
|
47
47
|
flags:
|
|
48
48
|
--budget <tokens> tighten or loosen the render budget (default 500,
|
|
@@ -111,6 +111,21 @@ const noContent = (url, detail, hint = "; 'oc raw' has the page's markdown if th
|
|
|
111
111
|
process.exitCode = NO_CONTENT_EXIT;
|
|
112
112
|
};
|
|
113
113
|
|
|
114
|
+
/**
|
|
115
|
+
* Forget everything saved under a name, for `oc logout` and `oc session rm`.
|
|
116
|
+
* The page saved under a name can be the distilled text of a page only the
|
|
117
|
+
* cookies could reach, so the two go together: after either command nothing
|
|
118
|
+
* of that login remains. Cookies go first so a failure on the page file never
|
|
119
|
+
* leaves the credential behind.
|
|
120
|
+
* @param {string} name
|
|
121
|
+
* @returns {boolean} whether anything was on disk to forget
|
|
122
|
+
*/
|
|
123
|
+
function forgetSession(name) {
|
|
124
|
+
const hadJar = clearCookieJar(name);
|
|
125
|
+
const hadPage = clearSession(name);
|
|
126
|
+
return hadJar || hadPage;
|
|
127
|
+
}
|
|
128
|
+
|
|
114
129
|
const LOGIN_USAGE = 'usage: printf %s "session=..." | oc login --cookie - --domain example.com'
|
|
115
130
|
+ ' [--expires 1h] [--session name] [--allow-http]';
|
|
116
131
|
|
|
@@ -208,11 +223,7 @@ async function main() {
|
|
|
208
223
|
}
|
|
209
224
|
|
|
210
225
|
if (command === 'logout') {
|
|
211
|
-
|
|
212
|
-
clearCookieJar(name);
|
|
213
|
-
// The page saved under this name can be the distilled text of a page only
|
|
214
|
-
// the cookies could reach, so logout drops it too.
|
|
215
|
-
clearSession(name);
|
|
226
|
+
forgetSession(args[0] ? assertSafeName(args[0]) : sessionName);
|
|
216
227
|
return;
|
|
217
228
|
}
|
|
218
229
|
|
|
@@ -365,7 +376,31 @@ async function main() {
|
|
|
365
376
|
case 'submit': return act.submit(args[0] ? Number(args[0]) : undefined);
|
|
366
377
|
case 'back': return act.back();
|
|
367
378
|
case 'sites': return console.log(listSites());
|
|
368
|
-
case 'session':
|
|
379
|
+
case 'session': {
|
|
380
|
+
// Saved sessions accumulate on disk (one JSON per page kept for
|
|
381
|
+
// do/read/next), so agents can inspect and drop them without guessing
|
|
382
|
+
// paths under ~/.only-cli. With no name, rm targets --session.
|
|
383
|
+
const [sub, target] = args;
|
|
384
|
+
if (sub === 'ls') {
|
|
385
|
+
const list = listSessions();
|
|
386
|
+
if (values.json) return console.log(JSON.stringify(list));
|
|
387
|
+
if (!list.length) return console.log('no saved sessions');
|
|
388
|
+
for (const s of list) {
|
|
389
|
+
console.log(`${s.name}${s.url ? ` ${s.url}` : ''}${s.title ? ` (${s.title})` : ''}${s.cookies ? ' [cookies]' : ''}`);
|
|
390
|
+
}
|
|
391
|
+
return;
|
|
392
|
+
}
|
|
393
|
+
if (sub === 'rm') {
|
|
394
|
+
const name = target ? assertSafeName(target) : sessionName;
|
|
395
|
+
// A name nothing was saved under is most likely a typo, and an agent
|
|
396
|
+
// that reads "forgot session" would move on believing the login is
|
|
397
|
+
// gone, so rm fails loud instead of succeeding at nothing.
|
|
398
|
+
if (!forgetSession(name)) throw new Error(`no such session '${name}', run oc session ls`);
|
|
399
|
+
if (values.json) return console.log(JSON.stringify({ forgotten: name }));
|
|
400
|
+
return console.log(`forgot session '${name}'`);
|
|
401
|
+
}
|
|
402
|
+
throw new Error(`usage: oc session ls|rm [name]`);
|
|
403
|
+
}
|
|
369
404
|
default:
|
|
370
405
|
throw new Error(`unknown command '${command}', run oc --help`);
|
|
371
406
|
}
|
package/src/cookies.js
CHANGED
|
@@ -7,7 +7,7 @@ import net from 'node:net';
|
|
|
7
7
|
import { join } from 'node:path';
|
|
8
8
|
import { mkdirSync, readFileSync, writeFileSync, unlinkSync, readdirSync, chmodSync } from 'node:fs';
|
|
9
9
|
|
|
10
|
-
import { sessionDir, assertSafeName } from './session.js';
|
|
10
|
+
import { sessionDir, assertSafeName, COOKIE_JAR_SUFFIX } from './session.js';
|
|
11
11
|
|
|
12
12
|
const DEFAULT_EXPIRES_MS = 60 * 60 * 1000; // 1h
|
|
13
13
|
export { DEFAULT_EXPIRES_MS };
|
|
@@ -53,7 +53,7 @@ function clip(value) {
|
|
|
53
53
|
* @returns {string}
|
|
54
54
|
*/
|
|
55
55
|
export function cookieJarPath(name) {
|
|
56
|
-
return join(sessionDir(), `${assertSafeName(name)}
|
|
56
|
+
return join(sessionDir(), `${assertSafeName(name)}${COOKIE_JAR_SUFFIX}`);
|
|
57
57
|
}
|
|
58
58
|
|
|
59
59
|
/**
|
|
@@ -265,13 +265,18 @@ export function saveCookieJar(name, jar) {
|
|
|
265
265
|
}
|
|
266
266
|
|
|
267
267
|
/**
|
|
268
|
+
* Only a missing jar is fine to ignore: any other failure leaves a live
|
|
269
|
+
* credential on disk that the caller may be about to report as gone.
|
|
268
270
|
* @param {string} name
|
|
271
|
+
* @returns {boolean} whether a jar was removed
|
|
269
272
|
*/
|
|
270
273
|
export function clearCookieJar(name) {
|
|
271
274
|
try {
|
|
272
275
|
unlinkSync(cookieJarPath(name));
|
|
273
|
-
|
|
274
|
-
|
|
276
|
+
return true;
|
|
277
|
+
} catch (err) {
|
|
278
|
+
if (err?.code === 'ENOENT') return false;
|
|
279
|
+
throw err;
|
|
275
280
|
}
|
|
276
281
|
}
|
|
277
282
|
|
|
@@ -287,12 +292,13 @@ export function purgeExpiredJars() {
|
|
|
287
292
|
return;
|
|
288
293
|
}
|
|
289
294
|
for (const file of readdirSync(dir)) {
|
|
290
|
-
if (!file.endsWith(
|
|
291
|
-
const name = file.slice(0, -'.cookies.json'.length);
|
|
295
|
+
if (!file.endsWith(COOKIE_JAR_SUFFIX)) continue;
|
|
292
296
|
try {
|
|
293
297
|
const jar = /** @type {CookieJar} */ (JSON.parse(readFileSync(join(dir, file), 'utf8')));
|
|
294
|
-
if (isSessionExpired(jar))
|
|
298
|
+
if (isSessionExpired(jar)) unlinkSync(join(dir, file));
|
|
295
299
|
} catch {
|
|
300
|
+
// A jar that will not parse is as useless as an expired one, and a sweep
|
|
301
|
+
// that cannot delete a file has nothing to report: the next one retries.
|
|
296
302
|
try { unlinkSync(join(dir, file)); } catch {}
|
|
297
303
|
}
|
|
298
304
|
}
|
package/src/distill.js
CHANGED
|
@@ -504,9 +504,15 @@ export function feedToHTML(text) {
|
|
|
504
504
|
// HTML itself, ready to be embedded and parsed like any page.
|
|
505
505
|
const body = (entry.querySelector('content') ?? entry.querySelector('summary') ?? entry.querySelector('description'))?.textContent ?? '';
|
|
506
506
|
parts.push('<article>');
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
507
|
+
// The title is the link. It used to sit beside the byline as an anchor
|
|
508
|
+
// labelled "open", the same label on every entry, and the repeated-controls
|
|
509
|
+
// filter hid the lot as chrome, so nothing in a subreddit feed led to a
|
|
510
|
+
// post: `do <n>` on one read its heading instead of following it (#59). A
|
|
511
|
+
// heading that is exactly one anchor becomes a followable heading in the
|
|
512
|
+
// walk, so the number the agent already sees is the one that opens it.
|
|
513
|
+
if (title) parts.push(`<h2>${href ? `<a href="${esc(href)}">${esc(title)}</a>` : esc(title)}</h2>`);
|
|
514
|
+
if (byline || (href && !title)) {
|
|
515
|
+
parts.push(`<p>${esc(byline)}${href && !title ? ` <a href="${esc(href)}">open</a>` : ''}</p>`);
|
|
510
516
|
}
|
|
511
517
|
parts.push(body, '</article>');
|
|
512
518
|
}
|
package/src/fetch.js
CHANGED
|
@@ -497,7 +497,10 @@ export function proxyGet(url, proxy, headers = {}, tlsOpts = {}) {
|
|
|
497
497
|
* identity got the page (impers:chrome, impers:firefox, or fetch)
|
|
498
498
|
*/
|
|
499
499
|
export async function fetchPage(url, { jar } = {}) {
|
|
500
|
-
|
|
500
|
+
let target = /^https?:\/\//i.test(url) ? url : `https://${url}`;
|
|
501
|
+
// A reader with no cookies for reddit.com gets the feed where the page
|
|
502
|
+
// would be a login wall; one who logged in gets the page it asked for.
|
|
503
|
+
if (!jar?.cookieHeaderFor(target)) target = redditFeedURL(target) ?? target;
|
|
501
504
|
await assertSafeTarget(target);
|
|
502
505
|
const impers = await loadImpers();
|
|
503
506
|
return impers ? viaImpers(impers, target, jar) : viaFetch(target, jar);
|
|
@@ -555,6 +558,62 @@ function captureSetCookie(jar, url, res) {
|
|
|
555
558
|
jar.storeFromResponse(url, getSetCookieHeaders(res));
|
|
556
559
|
}
|
|
557
560
|
|
|
561
|
+
// Hosts whose edge answers the chrome fingerprint with a 403 or a 429 while
|
|
562
|
+
// letting firefox through. reddit.com started doing this in 2026 (#52), so
|
|
563
|
+
// starting with chrome there would turn every read into two requests against
|
|
564
|
+
// a per-address rate limit of about ten a minute. Subdomains inherit the
|
|
565
|
+
// entry.
|
|
566
|
+
const FIREFOX_FIRST_HOSTS = ['reddit.com'];
|
|
567
|
+
|
|
568
|
+
// Reddit has sent logged-out readers of its HTML pages to a login page since
|
|
569
|
+
// June 2026 (#52), while the Atom feed beside each of those pages still
|
|
570
|
+
// answers. A feed entry links to the page, so following a post out of a
|
|
571
|
+
// subreddit feed used to land on the wall. The front page, subreddit
|
|
572
|
+
// listings, posts, user pages and search are mapped to their feeds here;
|
|
573
|
+
// anything else on reddit.com is fetched as asked.
|
|
574
|
+
const REDDIT_HOSTS = ['reddit.com', 'www.reddit.com', 'old.reddit.com', 'new.reddit.com', 'np.reddit.com'];
|
|
575
|
+
const SEG = '[A-Za-z0-9_.-]+';
|
|
576
|
+
const REDDIT_FEED_PATHS = new RegExp(
|
|
577
|
+
`^(?:|/r/${SEG}(?:/(?:new|top|hot|rising))?|/(?:r/${SEG}/)?comments/${SEG}(?:/${SEG}){0,2}|/u(?:ser)?/${SEG})$`,
|
|
578
|
+
);
|
|
579
|
+
|
|
580
|
+
/**
|
|
581
|
+
* The www.reddit.com Atom feed for a reddit.com page URL, or null when the URL
|
|
582
|
+
* is not one of the page shapes that has a feed, or is a feed already.
|
|
583
|
+
* @param {string} target
|
|
584
|
+
* @returns {string | null}
|
|
585
|
+
*/
|
|
586
|
+
export function redditFeedURL(target) {
|
|
587
|
+
let u;
|
|
588
|
+
try {
|
|
589
|
+
u = new URL(target);
|
|
590
|
+
} catch {
|
|
591
|
+
return null;
|
|
592
|
+
}
|
|
593
|
+
if (!REDDIT_HOSTS.includes(u.hostname.toLowerCase())) return null;
|
|
594
|
+
const path = u.pathname.replace(/\/+$/, '');
|
|
595
|
+
if (/\.(?:rss|json|xml)$/i.test(path)) return null;
|
|
596
|
+
if (path === '/search') return `https://www.reddit.com/search.rss${u.search}`;
|
|
597
|
+
if (!REDDIT_FEED_PATHS.test(path)) return null;
|
|
598
|
+
return `https://www.reddit.com${path.replace(/^\/u\//, '/user/')}/.rss${u.search}`;
|
|
599
|
+
}
|
|
600
|
+
|
|
601
|
+
/**
|
|
602
|
+
* The order in which impers identities are tried for a URL.
|
|
603
|
+
* @param {string} target
|
|
604
|
+
* @returns {['chrome', 'firefox'] | ['firefox', 'chrome']}
|
|
605
|
+
*/
|
|
606
|
+
export function identityOrder(target) {
|
|
607
|
+
let host = '';
|
|
608
|
+
try {
|
|
609
|
+
host = new URL(target).hostname.toLowerCase();
|
|
610
|
+
} catch {
|
|
611
|
+
return ['chrome', 'firefox'];
|
|
612
|
+
}
|
|
613
|
+
const firefoxFirst = FIREFOX_FIRST_HOSTS.some((h) => host === h || host.endsWith(`.${h}`));
|
|
614
|
+
return firefoxFirst ? ['firefox', 'chrome'] : ['chrome', 'firefox'];
|
|
615
|
+
}
|
|
616
|
+
|
|
558
617
|
/**
|
|
559
618
|
* Fetch a page through impers, downgrading identity when one is refused.
|
|
560
619
|
* Exported so the downgrade chain can be proven against a fake impers; the
|
|
@@ -565,14 +624,15 @@ function captureSetCookie(jar, url, res) {
|
|
|
565
624
|
* @returns {Promise<{url: string, html: string, status: number, via: string}>}
|
|
566
625
|
*/
|
|
567
626
|
export async function viaImpers(impers, target, jar) {
|
|
568
|
-
//
|
|
569
|
-
// blocked first attempt gets one cheap retry with a second identity. An
|
|
627
|
+
// A blocked first attempt gets one cheap retry with the other identity. An
|
|
570
628
|
// ImpersonateError is the same story one layer down: impers resolves the
|
|
571
629
|
// 'chrome' alias to its newest fingerprint, but the native library it loads
|
|
572
630
|
// can be an older system copy of libcurl-impersonate that predates that
|
|
573
631
|
// fingerprint and refuses it before any request leaves. Firefox aliases to
|
|
574
632
|
// an older target that such a library usually still knows, and when both
|
|
575
633
|
// identities are refused the plain fetch transport still gets the page.
|
|
634
|
+
// Hosts that are known to refuse chrome outright start with firefox, so the
|
|
635
|
+
// usual case there costs one request instead of a 403 and a retry.
|
|
576
636
|
const asking = (impersonate) => (url) =>
|
|
577
637
|
impers.get(url, {
|
|
578
638
|
impersonate,
|
|
@@ -590,11 +650,12 @@ export async function viaImpers(impers, target, jar) {
|
|
|
590
650
|
return null;
|
|
591
651
|
}
|
|
592
652
|
};
|
|
593
|
-
|
|
594
|
-
let
|
|
653
|
+
const [first, second] = identityOrder(target);
|
|
654
|
+
let via = `impers:${first}`;
|
|
655
|
+
let got = await attempt(first);
|
|
595
656
|
if (!got || got.status >= 400) {
|
|
596
|
-
via =
|
|
597
|
-
got = (await attempt(
|
|
657
|
+
via = `impers:${second}`;
|
|
658
|
+
got = (await attempt(second)) ?? got;
|
|
598
659
|
}
|
|
599
660
|
if (!got) return viaFetch(target, jar);
|
|
600
661
|
const { res, status } = got;
|
package/src/session.js
CHANGED
|
@@ -12,10 +12,16 @@
|
|
|
12
12
|
|
|
13
13
|
import { homedir } from 'node:os';
|
|
14
14
|
import { join } from 'node:path';
|
|
15
|
-
import { mkdirSync, readFileSync, writeFileSync, chmodSync, unlinkSync } from 'node:fs';
|
|
15
|
+
import { mkdirSync, readFileSync, writeFileSync, chmodSync, unlinkSync, readdirSync } from 'node:fs';
|
|
16
16
|
|
|
17
17
|
export const DEFAULT_SESSION = 'default';
|
|
18
18
|
|
|
19
|
+
// The cookie sidecar sits next to the page file as `<name>.cookies.json`
|
|
20
|
+
// (see cookies.js). Both modules split filenames on this suffix, so it lives
|
|
21
|
+
// in one place: a rename that reached only one of them would make `session
|
|
22
|
+
// ls` list jars as pages, and `session rm` unlink a jar as a page.
|
|
23
|
+
export const COOKIE_JAR_SUFFIX = '.cookies.json';
|
|
24
|
+
|
|
19
25
|
// OC_HOME relocates the whole state directory, for sandboxes, CI, and tests.
|
|
20
26
|
export const sessionDir = () => join(process.env.OC_HOME ?? join(homedir(), '.only-cli'), 'sessions');
|
|
21
27
|
|
|
@@ -25,13 +31,19 @@ export const sessionDir = () => join(process.env.OC_HOME ?? join(homedir(), '.on
|
|
|
25
31
|
// the store. Names are user-facing labels, so this charset loses nothing real.
|
|
26
32
|
const SAFE_NAME = /^[A-Za-z0-9._-]+$/;
|
|
27
33
|
|
|
34
|
+
// A name ending in '.cookies' would save its page at `<x>.cookies.json`, the
|
|
35
|
+
// path of session `<x>`'s cookie jar, so `oc logout x` would delete it and
|
|
36
|
+
// `oc session ls` would hide it.
|
|
37
|
+
const isSafeName = (name) => typeof name === 'string' && name !== '.' && name !== '..'
|
|
38
|
+
&& !name.endsWith('.cookies') && SAFE_NAME.test(name);
|
|
39
|
+
|
|
28
40
|
/**
|
|
29
41
|
* @param {string} name
|
|
30
42
|
* @returns {string} the same name, once it is known to be a safe filename
|
|
31
43
|
*/
|
|
32
44
|
export function assertSafeName(name) {
|
|
33
|
-
if (
|
|
34
|
-
throw new Error(`invalid session name '${name}', use letters, numbers, '.', '-', or '_'`);
|
|
45
|
+
if (!isSafeName(name)) {
|
|
46
|
+
throw new Error(`invalid session name '${name}', use letters, numbers, '.', '-', or '_' (not ending in '.cookies')`);
|
|
35
47
|
}
|
|
36
48
|
return name;
|
|
37
49
|
}
|
|
@@ -167,14 +179,19 @@ export function saveSession(name, state) {
|
|
|
167
179
|
* Drop a saved page. `oc logout` calls this alongside clearing the cookie jar:
|
|
168
180
|
* a snapshot taken under a login holds that page's text, so leaving it behind
|
|
169
181
|
* would make logout mean "the cookies are gone" rather than "nothing of this
|
|
170
|
-
* login remains".
|
|
182
|
+
* login remains". Only a missing file is fine to ignore: a permission error
|
|
183
|
+
* or a name that is a directory leaves the page on disk, and the caller is
|
|
184
|
+
* about to tell the user it is gone.
|
|
171
185
|
* @param {string} name
|
|
186
|
+
* @returns {boolean} whether a saved page was removed
|
|
172
187
|
*/
|
|
173
188
|
export function clearSession(name) {
|
|
174
189
|
try {
|
|
175
190
|
unlinkSync(sessionPath(name));
|
|
176
|
-
|
|
177
|
-
|
|
191
|
+
return true;
|
|
192
|
+
} catch (err) {
|
|
193
|
+
if (err?.code === 'ENOENT') return false;
|
|
194
|
+
throw err;
|
|
178
195
|
}
|
|
179
196
|
}
|
|
180
197
|
|
|
@@ -191,3 +208,43 @@ export function loadSession(name) {
|
|
|
191
208
|
return null;
|
|
192
209
|
}
|
|
193
210
|
}
|
|
211
|
+
|
|
212
|
+
/**
|
|
213
|
+
* Every session on disk, for `oc session ls`: a name is a session when it has
|
|
214
|
+
* a saved page, a cookie jar, or both, since `oc login` creates a jar without
|
|
215
|
+
* a page and that credential is exactly what an agent auditing leftover logins
|
|
216
|
+
* needs to see. An unreadable page is still listed by name: `oc session rm`
|
|
217
|
+
* can drop it. Only names oc itself could have written are listed, so a stray
|
|
218
|
+
* `..json` or a directory named like a page never shows up as something rm
|
|
219
|
+
* then cannot remove.
|
|
220
|
+
* @returns {{name: string, url: string|null, title: string|null, savedAt: string|null, cookies: boolean}[]}
|
|
221
|
+
*/
|
|
222
|
+
export function listSessions() {
|
|
223
|
+
let entries;
|
|
224
|
+
try {
|
|
225
|
+
entries = readdirSync(sessionDir(), { withFileTypes: true });
|
|
226
|
+
} catch {
|
|
227
|
+
return [];
|
|
228
|
+
}
|
|
229
|
+
/** @type {Map<string, {name: string, url: string|null, title: string|null, savedAt: string|null, cookies: boolean}>} */
|
|
230
|
+
const byName = new Map();
|
|
231
|
+
const entry = (name) => {
|
|
232
|
+
if (!byName.has(name)) byName.set(name, { name, url: null, title: null, savedAt: null, cookies: false });
|
|
233
|
+
return byName.get(name);
|
|
234
|
+
};
|
|
235
|
+
for (const file of entries) {
|
|
236
|
+
if (!file.isFile()) continue;
|
|
237
|
+
const jarName = file.name.endsWith(COOKIE_JAR_SUFFIX) ? file.name.slice(0, -COOKIE_JAR_SUFFIX.length) : null;
|
|
238
|
+
const pageName = jarName == null && file.name.endsWith('.json') ? file.name.slice(0, -'.json'.length) : null;
|
|
239
|
+
if (jarName != null && isSafeName(jarName)) {
|
|
240
|
+
entry(jarName).cookies = true;
|
|
241
|
+
} else if (pageName != null && isSafeName(pageName)) {
|
|
242
|
+
const info = entry(pageName);
|
|
243
|
+
const state = loadSession(pageName);
|
|
244
|
+
info.url = state?.url ?? null;
|
|
245
|
+
info.title = state?.title ?? null;
|
|
246
|
+
info.savedAt = state?.savedAt ?? null;
|
|
247
|
+
}
|
|
248
|
+
}
|
|
249
|
+
return [...byName.values()].sort((a, b) => a.name.localeCompare(b.name));
|
|
250
|
+
}
|