@only-cli/oc 0.3.0-beta.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -2
- package/package.json +1 -1
- package/skills/web-browsing-cli/SKILL.md +11 -9
- package/src/act.js +42 -7
- package/src/cli.js +4 -2
- package/src/distill.js +170 -19
- package/src/fetch.js +51 -34
- package/src/render.js +26 -3
package/README.md
CHANGED
|
@@ -59,7 +59,8 @@ No setup at all also works: `npx @only-cli/oc` runs without a global install, an
|
|
|
59
59
|
```
|
|
60
60
|
oc open <url> fetch and render a page with numbered actions
|
|
61
61
|
oc do <n> follow the numbered link [n], or read [n] if it is text
|
|
62
|
-
oc find <query> where a string appears on the page already open
|
|
62
|
+
oc find <query> where a string appears on the page already open, or
|
|
63
|
+
the region itself when only one place matches
|
|
63
64
|
oc read <n> full text of the region at [n], up to 2000 tokens
|
|
64
65
|
oc next the next budget worth of the page already open
|
|
65
66
|
oc raw [url] distilled markdown of the whole page
|
|
@@ -69,7 +70,7 @@ oc submit [n] submit a form (planned)
|
|
|
69
70
|
|
|
70
71
|
Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTML), `--session <name>`, `--verbose`/`-v` (metrics on stderr, or export `OC_VERBOSE=1`).
|
|
71
72
|
|
|
72
|
-
`oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
|
|
73
|
+
`oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. A result title on a search page is a link, so `oc do` on it opens the result rather than repeating the title. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page, and a `find` with a single match prints that region instead of the number to read it with. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
|
|
73
74
|
|
|
74
75
|
## Supported websites
|
|
75
76
|
|
package/package.json
CHANGED
|
@@ -8,12 +8,13 @@ description: Token-efficient web browsing and web content extraction for AI agen
|
|
|
8
8
|
Renders a web page as a compact, numbered terminal view instead of raw HTML. A typical page is under 500 tokens.
|
|
9
9
|
|
|
10
10
|
```
|
|
11
|
-
npx --yes @only-cli/oc@0.
|
|
12
|
-
npx --yes @only-cli/oc@0.
|
|
13
|
-
npx --yes @only-cli/oc@0.
|
|
14
|
-
|
|
15
|
-
npx --yes @only-cli/oc@0.
|
|
16
|
-
npx --yes @only-cli/oc@0.
|
|
11
|
+
npx --yes @only-cli/oc@0.3.0 open <url> compact view, numbered elements
|
|
12
|
+
npx --yes @only-cli/oc@0.3.0 do <n> follow link [n], or read it if [n] is text
|
|
13
|
+
npx --yes @only-cli/oc@0.3.0 find <query> where a string appears, or that place itself
|
|
14
|
+
when only one matches
|
|
15
|
+
npx --yes @only-cli/oc@0.3.0 next next ~500 tokens of the page already open
|
|
16
|
+
npx --yes @only-cli/oc@0.3.0 read <n> full text of region [n]
|
|
17
|
+
npx --yes @only-cli/oc@0.3.0 raw [url] whole page as markdown (--html for cleaned HTML)
|
|
17
18
|
```
|
|
18
19
|
|
|
19
20
|
None of these except `open`/`do`/`raw <url>` fetch anything — they replay the page `open` already saved.
|
|
@@ -23,13 +24,14 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
|
|
|
23
24
|
- Line 1 is the title, then main content (article/thread/results); nav/sidebar/footer follow after `--- rest of page ---`, still numbered.
|
|
24
25
|
- `--- repeated controls hidden ---`: per-item chrome (save/report/reply) dropped as repetitive; `raw` keeps it.
|
|
25
26
|
- `[n]` marks a link, button, input, heading, or a text block long enough to be cut.
|
|
26
|
-
-
|
|
27
|
+
- Code blocks arrive as the page wrote them, lines and indentation intact, so a command in one can be run as printed.
|
|
28
|
+
- `... +820 chars`: block was cut there; `read <n>` prints it whole. The cut lands on the end of a sentence, or of a line in code, so what is shown is never half of one.
|
|
27
29
|
- `... 164 more blocks (~7,100 tokens)`: rest of page past budget — a cost estimate, not a fetch. Omitted when the page would finish only a little over budget; then it's printed whole instead.
|
|
28
30
|
- `actions:` footer lists valid next commands.
|
|
29
31
|
|
|
30
32
|
## Going further, cheapest first
|
|
31
33
|
|
|
32
|
-
- `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit.
|
|
34
|
+
- `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit. When one place matches, or when the matches all fit, it prints them in full: no `read <n>` afterwards.
|
|
33
35
|
- `read <n>` — one region in full: the block at `[n]` plus a little context, or the whole section for a heading.
|
|
34
36
|
- `next` — continues the same page from where the budget stopped.
|
|
35
37
|
- `raw` — everything, ~10x the cost. Use only when you need the whole page, not to hunt for a link's URL (use `do` for that).
|
|
@@ -42,7 +44,7 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
|
|
|
42
44
|
- `[6-9] 4 similar links` markers still work despite the collapsed text.
|
|
43
45
|
- Search result links resolve to the destination, not the tracking redirect.
|
|
44
46
|
- `do` on an input/button reports that instead (typing/submitting not yet supported).
|
|
45
|
-
- `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow.
|
|
47
|
+
- `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow. A heading that is itself a link, which is what a search result title is, opens instead.
|
|
46
48
|
- `--session <name>` keeps separate page state, for working on two sites at once.
|
|
47
49
|
|
|
48
50
|
## Flags
|
package/src/act.js
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
*/
|
|
10
10
|
|
|
11
11
|
import { DEFAULT_SESSION, handleFor, handleNumbers, loadSession, saveSession } from './session.js';
|
|
12
|
-
import { estimateTokens, formatBlock, render } from './render.js';
|
|
12
|
+
import { FINISH, estimateTokens, formatBlock, render } from './render.js';
|
|
13
13
|
|
|
14
14
|
export class NotImplemented extends Error {
|
|
15
15
|
constructor(command) {
|
|
@@ -63,7 +63,11 @@ export function activate(n, { session = DEFAULT_SESSION } = {}) {
|
|
|
63
63
|
const range = nums.length ? `1-${Math.max(...nums)}` : 'none';
|
|
64
64
|
throw new Error(`no [${n}] on ${state.url} (handles ${range}), run 'oc open <url>' again to renumber`);
|
|
65
65
|
}
|
|
66
|
-
|
|
66
|
+
// A heading can be a link: on a search results page the result title is one,
|
|
67
|
+
// and opening it is what `do` was asked for. Reading the title back instead
|
|
68
|
+
// cost a turn, and then another to find the number that does navigate, so a
|
|
69
|
+
// heading that has an href falls through to the link below.
|
|
70
|
+
if ((handle.type === 'text' || handle.type === 'heading') && !handle.href) {
|
|
67
71
|
// There is nothing to follow, but the agent asked to see what is at [n],
|
|
68
72
|
// and that is what read prints. Refusing would spend a whole turn to name
|
|
69
73
|
// the command that should have run, and a turn costs more than the page.
|
|
@@ -208,14 +212,41 @@ export function find(query, { session = DEFAULT_SESSION, budget = 500 } = {}) {
|
|
|
208
212
|
return `no match for "${query}"${tried} in ${blocks.length} blocks on ${state.url}, try fewer words or 'oc raw' for the full text`;
|
|
209
213
|
}
|
|
210
214
|
|
|
211
|
-
const
|
|
215
|
+
const separately = loose ? ', matching the words separately' : '';
|
|
216
|
+
|
|
217
|
+
// One match is not an index, it is the answer. Naming the number and
|
|
218
|
+
// stopping spends a turn to say where to look, and the agent's next command
|
|
219
|
+
// is always the `read` that looks, so find does the reading. Measured on an
|
|
220
|
+
// AWS CLI reference page, `find "Example 7"` printed a 24 token heading and
|
|
221
|
+
// the example it names was in the block after it.
|
|
222
|
+
if (hits.length === 1 && hits[0].n != null) {
|
|
223
|
+
const only = hits[0];
|
|
224
|
+
const follow = only.type === 'link' ? 'do <n> | ' : '';
|
|
225
|
+
return [
|
|
226
|
+
`1 match for "${query}"${separately}, region [${only.n}]`,
|
|
227
|
+
read(only.n, { session, budget: budget * FINISH }),
|
|
228
|
+
`actions: ${follow}read <n> | next | raw`,
|
|
229
|
+
].join('\n');
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const lines = [`${hits.length} ${hits.length === 1 ? 'match' : 'matches'} for "${query}"${separately}`];
|
|
233
|
+
// A snippet is short because many matches have to share one screen. When
|
|
234
|
+
// every match would fit whole inside the allowance a page already gets for
|
|
235
|
+
// finishing (render's FINISH), showing them whole makes this command the
|
|
236
|
+
// answer instead of an index into a `read <n>` that has to run next. The
|
|
237
|
+
// trade is the same one FINISH documents, and it is not close: a few
|
|
238
|
+
// hundred tokens against a turn.
|
|
239
|
+
const label = (hit, text) => `[${hit.n ?? '?'}] ${text}`;
|
|
240
|
+
const whole = hits.reduce((n, h) => n + estimateTokens(label(h, h.text)) + 1, estimateTokens(lines[0]));
|
|
241
|
+
const full = whole <= budget * FINISH;
|
|
242
|
+
const cap = full ? budget * FINISH : budget;
|
|
212
243
|
let spent = estimateTokens(lines[0]);
|
|
213
244
|
let shown = 0;
|
|
214
245
|
let hasLinks = false;
|
|
215
246
|
for (const hit of hits) {
|
|
216
|
-
const line =
|
|
247
|
+
const line = label(hit, full ? hit.text : hit.snippet);
|
|
217
248
|
const cost = estimateTokens(line) + 1;
|
|
218
|
-
if (spent + cost >
|
|
249
|
+
if (spent + cost > cap && shown) break;
|
|
219
250
|
spent += cost;
|
|
220
251
|
shown++;
|
|
221
252
|
if (hit.type === 'link') hasLinks = true;
|
|
@@ -251,8 +282,12 @@ function search(blocks, terms) {
|
|
|
251
282
|
if (out.length && out.at(-1).n === n) continue;
|
|
252
283
|
const start = Math.max(0, Math.min(...found) - BEFORE);
|
|
253
284
|
const end = Math.min(block.text.length, start + SNIPPET);
|
|
254
|
-
|
|
255
|
-
|
|
285
|
+
// One line per match is the promise the list makes, and a code block is the
|
|
286
|
+
// one kind of block that carries lines of its own. They survive where they
|
|
287
|
+
// are read rather than indexed: the whole-match mode above, and `read`.
|
|
288
|
+
const window = block.text.slice(start, end).replace(/\n/g, ' ');
|
|
289
|
+
const snippet = `${start > 0 ? '... ' : ''}${window}${end < block.text.length ? ' ...' : ''}`;
|
|
290
|
+
out.push({ n, type: block.type, snippet, text: block.text });
|
|
256
291
|
}
|
|
257
292
|
return out;
|
|
258
293
|
}
|
package/src/cli.js
CHANGED
|
@@ -11,7 +11,8 @@ const HELP = `only-cli: the web as a compact terminal, built for AI agents.
|
|
|
11
11
|
usage: oc <command> [args] [flags]
|
|
12
12
|
|
|
13
13
|
open <url> fetch and render a page with numbered actions
|
|
14
|
-
find <query> where a string appears on the page already open
|
|
14
|
+
find <query> where a string appears on the page already open, or
|
|
15
|
+
the region itself when only one place matches
|
|
15
16
|
next the next budget worth of the page already open
|
|
16
17
|
read <n> full text of the region at [n], up to 2000 tokens
|
|
17
18
|
raw [url] distilled markdown of the whole page
|
|
@@ -36,7 +37,8 @@ flags:
|
|
|
36
37
|
|
|
37
38
|
'oc open' remembers the page it printed, so 'oc do 3' follows link [3] without
|
|
38
39
|
you ever handling its URL, and 'oc next' or 'oc read 12' picks up what the
|
|
39
|
-
budget left behind without fetching it again.
|
|
40
|
+
budget left behind without fetching it again. 'oc do' on a search result title
|
|
41
|
+
opens the result, because that title is a link. State lives in ~/.only-cli
|
|
40
42
|
(override with OC_HOME).`;
|
|
41
43
|
|
|
42
44
|
// A rendered page has to be remembered or its [3] means nothing to the next
|
package/src/distill.js
CHANGED
|
@@ -7,7 +7,7 @@ import TurndownService from 'turndown';
|
|
|
7
7
|
* @property {string} text
|
|
8
8
|
* @property {number} [n] - action handle
|
|
9
9
|
* @property {number} [level] - heading level 1..6
|
|
10
|
-
* @property {string} [href] - links
|
|
10
|
+
* @property {string} [href] - links, and a heading that is one
|
|
11
11
|
* @property {string} [name] - inputs only
|
|
12
12
|
*
|
|
13
13
|
* @typedef {Object} Page
|
|
@@ -32,6 +32,32 @@ const DROP = new Set([
|
|
|
32
32
|
// them, so the search below refuses to descend into them.
|
|
33
33
|
const FURNITURE = new Set(['nav', 'header', 'footer', 'aside']);
|
|
34
34
|
|
|
35
|
+
// Controls a page puts inside its own code samples, and the selector that
|
|
36
|
+
// finds one. Node's API docs give every code block a copy button, a module
|
|
37
|
+
// toggle and a language label, so the text of the sample ends in
|
|
38
|
+
// `javascriptcopy` unless the toolbar holding them is left out of it.
|
|
39
|
+
const CODE_CONTROLS = new Set(['button', 'input', 'select', 'textarea', 'label']);
|
|
40
|
+
const CONTROL = [...CODE_CONTROLS].join(', ');
|
|
41
|
+
|
|
42
|
+
// Lines in a code block are kept, where every other kind of text has its
|
|
43
|
+
// whitespace collapsed. Joining them would put `// comment` in front of the
|
|
44
|
+
// statements that followed it, so a sample an agent could run would arrive
|
|
45
|
+
// commented out, and the shape of the wreck is invisible on one line. Blank
|
|
46
|
+
// runs and the indentation the whole block shares are the parts that carry no
|
|
47
|
+
// meaning, so those go.
|
|
48
|
+
const codeText = (raw) => {
|
|
49
|
+
const lines = raw.replace(/\r\n?/g, '\n').replace(/[^\S\n]+$/gm, '').split('\n');
|
|
50
|
+
while (lines.length && !lines[0].trim()) lines.shift();
|
|
51
|
+
while (lines.length && !lines[lines.length - 1].trim()) lines.pop();
|
|
52
|
+
const indent = lines
|
|
53
|
+
.filter((l) => l.trim())
|
|
54
|
+
.reduce((least, l) => Math.min(least, l.length - l.trimStart().length), Infinity);
|
|
55
|
+
return lines
|
|
56
|
+
.map((l) => (Number.isFinite(indent) ? l.slice(indent) : l).trimEnd())
|
|
57
|
+
.join('\n')
|
|
58
|
+
.replace(/\n{3,}/g, '\n\n');
|
|
59
|
+
};
|
|
60
|
+
|
|
35
61
|
// Elements that end a line on the page and so must end one here. Without this
|
|
36
62
|
// the text of six separate posts merges into a single block, because nothing
|
|
37
63
|
// between them survives distillation to keep them apart.
|
|
@@ -82,6 +108,29 @@ const clean = (s) => s.replace(/\s+/g, ' ').trim();
|
|
|
82
108
|
const asHTML = (text, url = '', opts = {}) =>
|
|
83
109
|
jsonToHTML(text, url, opts) ?? youtubeToHTML(text) ?? transcriptToHTML(text) ?? feedToHTML(text) ?? text;
|
|
84
110
|
|
|
111
|
+
/**
|
|
112
|
+
* The link a heading is, if it is one. A search engine puts the result title
|
|
113
|
+
* in an anchor inside an <h2>, so a heading can be the most followable thing
|
|
114
|
+
* on the page, and taking only its text threw that away.
|
|
115
|
+
*
|
|
116
|
+
* The heading has to BE the link, not merely contain one: exactly one anchor,
|
|
117
|
+
* labelled with the whole heading. Documentation fails that test on purpose.
|
|
118
|
+
* Every heading in the Rust book and every one on an AWS CLI reference page
|
|
119
|
+
* carries a permalink to its own id, so following those would refetch the
|
|
120
|
+
* page the agent is already reading, which is worse than the reading it
|
|
121
|
+
* already gets. A bare fragment is never a destination.
|
|
122
|
+
* @param {any} node - the heading element
|
|
123
|
+
* @param {string} text - its cleaned text
|
|
124
|
+
* @returns {string|null}
|
|
125
|
+
*/
|
|
126
|
+
function headingHref(node, text) {
|
|
127
|
+
const anchors = node.querySelectorAll('a[href]');
|
|
128
|
+
if (anchors.length !== 1) return null;
|
|
129
|
+
const href = anchors[0].getAttribute('href') ?? '';
|
|
130
|
+
if (!href || href.startsWith('#')) return null;
|
|
131
|
+
return clean(anchors[0].textContent) === text ? href : null;
|
|
132
|
+
}
|
|
133
|
+
|
|
85
134
|
/**
|
|
86
135
|
* Reduce raw HTML to an interaction tree: readable text plus numbered
|
|
87
136
|
* elements, in document order. The walk is deterministic and numbering is a
|
|
@@ -105,6 +154,45 @@ export function distill(html, url = '') {
|
|
|
105
154
|
/** Subtree already emitted, skipped when the rest of the page is walked. */
|
|
106
155
|
let done = null;
|
|
107
156
|
|
|
157
|
+
/**
|
|
158
|
+
* The text of a code subtree, read as one string.
|
|
159
|
+
*
|
|
160
|
+
* A syntax highlighter gives every token its own element, so `s3://bucket/`
|
|
161
|
+
* reaches the walk as `s3`, `:`, `//`, `bucket`, `/`, and the rule that puts
|
|
162
|
+
* fragments back together only glues the ones that share a parent. The rest
|
|
163
|
+
* are space-joined, which turned an AWS example into
|
|
164
|
+
* `aws s3 cp s3 : // bucket / -- recursive`, a command an agent cannot run.
|
|
165
|
+
* Reading the subtree whole is what fixes that, and it discards nothing: over
|
|
166
|
+
* 172 pre elements on the AWS CLI reference, the Rust book, the Node API docs
|
|
167
|
+
* and the Python library docs, 159 are split this way and not one of them
|
|
168
|
+
* contains a link.
|
|
169
|
+
*
|
|
170
|
+
* textContent would be enough if pages put only code in their code blocks.
|
|
171
|
+
* A control inside one is chrome, and so is the block-level element holding
|
|
172
|
+
* it: that is how the toolbar's `javascript` label leaves with the copy
|
|
173
|
+
* button it sits beside. The test stays on block-level wrappers because a
|
|
174
|
+
* highlighter's own elements are inline, so a stray control can never take a
|
|
175
|
+
* line of code out with it.
|
|
176
|
+
* @param {any} node
|
|
177
|
+
* @returns {string}
|
|
178
|
+
*/
|
|
179
|
+
const verbatim = (node) => {
|
|
180
|
+
let out = '';
|
|
181
|
+
const gather = (n) => {
|
|
182
|
+
if (n.nodeType === 3) {
|
|
183
|
+
out += n.textContent ?? '';
|
|
184
|
+
return;
|
|
185
|
+
}
|
|
186
|
+
if (n.nodeType !== 1) return;
|
|
187
|
+
const tag = n.localName;
|
|
188
|
+
if (DROP.has(tag) || CODE_CONTROLS.has(tag) || hidden(n)) return;
|
|
189
|
+
if (n !== node && BLOCKY.has(tag) && n.querySelector(CONTROL)) return;
|
|
190
|
+
for (const child of n.childNodes) gather(child);
|
|
191
|
+
};
|
|
192
|
+
gather(node);
|
|
193
|
+
return out;
|
|
194
|
+
};
|
|
195
|
+
|
|
108
196
|
const walk = (node) => {
|
|
109
197
|
if (node.nodeType === 3) {
|
|
110
198
|
const raw = node.textContent ?? '';
|
|
@@ -126,7 +214,10 @@ export function distill(html, url = '') {
|
|
|
126
214
|
|
|
127
215
|
if (/^h[1-6]$/.test(tag)) {
|
|
128
216
|
const text = clean(node.textContent);
|
|
129
|
-
if (text)
|
|
217
|
+
if (text) {
|
|
218
|
+
const href = headingHref(node, text);
|
|
219
|
+
blocks.push({ type: 'heading', level: Number(tag[1]), text, ...(href ? { href } : {}) });
|
|
220
|
+
}
|
|
130
221
|
return;
|
|
131
222
|
}
|
|
132
223
|
if (tag === 'a' && node.getAttribute('href')) {
|
|
@@ -157,6 +248,21 @@ export function distill(html, url = '') {
|
|
|
157
248
|
if (text) blocks.push({ type: 'button', text });
|
|
158
249
|
return;
|
|
159
250
|
}
|
|
251
|
+
if (tag === 'pre' || tag === 'code') {
|
|
252
|
+
const raw = verbatim(node);
|
|
253
|
+
const text = tag === 'pre' ? codeText(raw) : clean(raw);
|
|
254
|
+
// A pre is its own line, which is what BLOCKY gave it before this branch
|
|
255
|
+
// started claiming it first. Inline code belongs to the sentence around
|
|
256
|
+
// it, so it carries the parent and the edge spacing a text node would and
|
|
257
|
+
// merges back into that sentence the same way.
|
|
258
|
+
const own = tag === 'pre';
|
|
259
|
+
if (own) blocks.push({ type: 'break' });
|
|
260
|
+
if (text) {
|
|
261
|
+
blocks.push({ type: 'text', text, host: node.parentNode, pre: /^\s/.test(raw), post: /\s$/.test(raw) });
|
|
262
|
+
}
|
|
263
|
+
if (own) blocks.push({ type: 'break' });
|
|
264
|
+
return;
|
|
265
|
+
}
|
|
160
266
|
if (BLOCKY.has(tag)) {
|
|
161
267
|
blocks.push({ type: 'break' });
|
|
162
268
|
for (const child of node.childNodes) walk(child);
|
|
@@ -513,6 +619,15 @@ const TITLE_KEYS = [
|
|
|
513
619
|
'full_name', 'summary', 'question', 'message',
|
|
514
620
|
];
|
|
515
621
|
|
|
622
|
+
// Keys an API is likely to put its list of results under. A response using one
|
|
623
|
+
// of these is a list whatever else it carries, so the name settles it before
|
|
624
|
+
// shape does: a sideloaded `included` array can outnumber the `items` the
|
|
625
|
+
// request was for without being what the request was for.
|
|
626
|
+
const CONTAINER_KEYS = [
|
|
627
|
+
'items', 'data', 'results', 'hits', 'records', 'rows', 'entries',
|
|
628
|
+
'nodes', 'edges', 'docs', 'list', 'children', 'values',
|
|
629
|
+
];
|
|
630
|
+
|
|
516
631
|
// Keys holding the item's own page. A URL under any other name is still found,
|
|
517
632
|
// by looking at values rather than names, but these win when several qualify.
|
|
518
633
|
const LINK_KEYS = ['link', 'url', 'html_url', 'web_url', 'permalink', 'href'];
|
|
@@ -540,6 +655,14 @@ const NESTED_PENALTY = 0.25;
|
|
|
540
655
|
// How many names the footer lists when it says which fields it left out.
|
|
541
656
|
const DROPPED_LISTED = 5;
|
|
542
657
|
|
|
658
|
+
// Characters of markup a field may carry before the compact view stops
|
|
659
|
+
// treating it as a document and starts treating it as somewhere to go. A
|
|
660
|
+
// question body arrives well under this and is worth rendering in place, links
|
|
661
|
+
// and code and all. A package readme arrives at four figures and distils into
|
|
662
|
+
// more blocks than the resource it is attached to has fields, which buries the
|
|
663
|
+
// resource the response was fetched for. `oc raw` renders either one in full.
|
|
664
|
+
const BODY_CAP = 4000;
|
|
665
|
+
|
|
543
666
|
// Seconds and milliseconds since the epoch, bounded either side so an ordinary
|
|
544
667
|
// count (a score, a byte size) is never mistaken for a date.
|
|
545
668
|
const EPOCH_S = [1e9, 4e9];
|
|
@@ -572,6 +695,10 @@ const decodeEntities = (s) =>
|
|
|
572
695
|
const isPlain = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
|
|
573
696
|
const isURL = (v) => typeof v === 'string' && /^https?:\/\/\S+$/.test(v);
|
|
574
697
|
const looksHTML = (v) => typeof v === 'string' && /<\/?(p|div|pre|code|br|ul|ol|li|h[1-6]|blockquote|table|img|a|em|strong)\b[^>]*>/i.test(v);
|
|
698
|
+
// Markup as one line of prose, for a body the compact view is pointing at
|
|
699
|
+
// rather than rendering. The distiller is what reads markup properly; this
|
|
700
|
+
// only has to make a line an agent can tell one body from another by.
|
|
701
|
+
const stripTags = (v) => clean(decodeEntities(String(v).replace(/<[^>]*>/g, ' ')));
|
|
575
702
|
|
|
576
703
|
/**
|
|
577
704
|
* One level of flattening, so `owner: {display_name}` becomes an
|
|
@@ -662,32 +789,40 @@ function chooseFields(rows, skip) {
|
|
|
662
789
|
}
|
|
663
790
|
|
|
664
791
|
/**
|
|
665
|
-
* Pick
|
|
666
|
-
* otherwise the
|
|
667
|
-
*
|
|
668
|
-
*
|
|
792
|
+
* Pick what the response is actually about: the root when it is an array or a
|
|
793
|
+
* single resource, otherwise the array of objects at the top level that holds
|
|
794
|
+
* the results. Everything beside it is metadata about the request rather than
|
|
795
|
+
* content.
|
|
669
796
|
* @param {any} data
|
|
670
797
|
* @returns {{items: any[], meta: Record<string, any>}}
|
|
671
798
|
*/
|
|
672
799
|
function mainArray(data) {
|
|
673
800
|
if (Array.isArray(data)) return { items: data, meta: {} };
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
let items = null;
|
|
801
|
+
/** @type {Map<string, any[]>} */
|
|
802
|
+
const arrays = new Map();
|
|
677
803
|
for (const [k, v] of Object.entries(data)) {
|
|
678
804
|
if (!Array.isArray(v) || !v.length) continue;
|
|
679
805
|
if (!v.some(isPlain)) continue;
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
806
|
+
arrays.set(k, v);
|
|
807
|
+
}
|
|
808
|
+
let key = CONTAINER_KEYS.find((k) => arrays.has(k)) ?? '';
|
|
809
|
+
// A root carrying its own name is the resource, and an array hanging off it
|
|
810
|
+
// describes that resource rather than being the subject in its place. Taking
|
|
811
|
+
// the longest array regardless titled the npm registry's package endpoint
|
|
812
|
+
// after its two maintainers and demoted the package to the metadata line,
|
|
813
|
+
// where a 9KB readme then cost more than the rest of the page put together.
|
|
814
|
+
const named = TITLE_KEYS.some((k) => typeof data[k] === 'string' && data[k].trim() !== '');
|
|
815
|
+
if (!key && !named) {
|
|
816
|
+
for (const [k, v] of arrays) {
|
|
817
|
+
if (!key || v.length > (arrays.get(key)?.length ?? 0)) key = k;
|
|
683
818
|
}
|
|
684
819
|
}
|
|
685
|
-
// A response with no array is a single resource, which
|
|
686
|
-
// rather than as a special case.
|
|
687
|
-
if (!
|
|
820
|
+
// A response with no results array of its own is a single resource, which
|
|
821
|
+
// renders as one item rather than as a special case.
|
|
822
|
+
if (!key) return { items: [data], meta: {} };
|
|
688
823
|
const meta = { ...data };
|
|
689
824
|
delete meta[key];
|
|
690
|
-
return { items, meta };
|
|
825
|
+
return { items: arrays.get(key) ?? [data], meta };
|
|
691
826
|
}
|
|
692
827
|
|
|
693
828
|
/**
|
|
@@ -784,9 +919,16 @@ export function jsonToHTML(text, url = '', { full = false } = {}) {
|
|
|
784
919
|
if (kept && !kept.has(key) && !looksHTML(value)) continue;
|
|
785
920
|
// A field carrying HTML is a page in itself: `filter=withbody` on the
|
|
786
921
|
// Stack Exchange API puts a whole question in one. It goes through the
|
|
787
|
-
// distiller like any other markup instead of into a cell
|
|
922
|
+
// distiller like any other markup instead of into a cell, unless it is
|
|
923
|
+
// longer than the compact view can afford, in which case it becomes one
|
|
924
|
+
// numbered line rather than a dozen blocks that bury the item it hangs
|
|
925
|
+
// off. `oc read <n>` opens it at a budget that fits it, `oc raw` always.
|
|
788
926
|
if (looksHTML(value)) {
|
|
789
|
-
|
|
927
|
+
if (full || String(value).length <= BODY_CAP) {
|
|
928
|
+
bodies.push(String(value));
|
|
929
|
+
continue;
|
|
930
|
+
}
|
|
931
|
+
long.push(`<p>${escHTML(`${key}: ${stripTags(value)}`)}</p>`);
|
|
790
932
|
continue;
|
|
791
933
|
}
|
|
792
934
|
const rendered = row.get(key);
|
|
@@ -824,12 +966,21 @@ export function jsonToHTML(text, url = '', { full = false } = {}) {
|
|
|
824
966
|
}
|
|
825
967
|
|
|
826
968
|
const metaBits = [];
|
|
969
|
+
const metaLong = [];
|
|
827
970
|
for (const [key, value] of Object.entries(meta)) {
|
|
828
971
|
if (value === null || typeof value === 'object') continue;
|
|
829
972
|
const rendered = renderValue(key, value);
|
|
830
|
-
if (rendered)
|
|
973
|
+
if (!rendered) continue;
|
|
974
|
+
// A summary line has to stay a line. One long scalar at the root, a
|
|
975
|
+
// package readme or an endpoint description, would otherwise spend the
|
|
976
|
+
// whole page budget here, so it becomes a block of its own instead. The
|
|
977
|
+
// block is numbered, so `oc read <n>` opens it when it fits that budget
|
|
978
|
+
// and `oc raw` has it whatever its size.
|
|
979
|
+
if (rendered.length > TEXT_CAP) metaLong.push(`<p>${escHTML(`${key}: ${rendered}`)}</p>`);
|
|
980
|
+
else metaBits.push(`${key}=${rendered}`);
|
|
831
981
|
}
|
|
832
982
|
if (metaBits.length) parts.push(`<p>response: ${escHTML(metaBits.join(', '))}</p>`);
|
|
983
|
+
parts.push(...metaLong);
|
|
833
984
|
|
|
834
985
|
const count = `${items.length} ${items.length === 1 ? 'item' : 'items'}`;
|
|
835
986
|
return `<html><head><title>${escHTML(jsonTitle(url, count))}</title></head><body>\n${parts.join('\n')}\n</body></html>`;
|
package/src/fetch.js
CHANGED
|
@@ -24,6 +24,25 @@ const loadImpers = () => {
|
|
|
24
24
|
const BLOCKED_MESSAGE = 'blocked: private or internal URL';
|
|
25
25
|
const MAX_REDIRECTS = 20;
|
|
26
26
|
|
|
27
|
+
// What oc can turn into text: any text/* type, plus the application/* types
|
|
28
|
+
// that are really text (json, xml, and the +json / +xml families a feed or an
|
|
29
|
+
// API answers with). A PNG matches none of these, and rendering one produces
|
|
30
|
+
// pages of mojibake an agent then pays for, so it is refused by name instead.
|
|
31
|
+
const READABLE_TYPE = /^\s*(?:text\/|application\/(?:json|xml|javascript|x-ndjson|[\w.+-]*\+(?:json|xml)))/i;
|
|
32
|
+
|
|
33
|
+
/**
|
|
34
|
+
* Refuse a response oc cannot read as text. Both transports call this: the
|
|
35
|
+
* gate has to live on whichever client got the page, or the same URL renders
|
|
36
|
+
* as an error through fetch and as binary noise through impers.
|
|
37
|
+
* @param {string | null | undefined} type - the content-type header
|
|
38
|
+
*/
|
|
39
|
+
export function assertReadableType(type) {
|
|
40
|
+
// No header at all is not a refusal: plenty of small servers omit it, and
|
|
41
|
+
// the distiller handles whatever comes back.
|
|
42
|
+
if (!type || READABLE_TYPE.test(type)) return;
|
|
43
|
+
throw new Error(`not a page oc can read (${type.split(';')[0].trim()}), it renders HTML, XML feeds, JSON, and plain text`);
|
|
44
|
+
}
|
|
45
|
+
|
|
27
46
|
// IPv4 ranges with no business receiving a server-initiated fetch: loopback,
|
|
28
47
|
// link-local, the three RFC 1918 private blocks, carrier-grade NAT, the
|
|
29
48
|
// unspecified/broadcast addresses, and the documentation/benchmark ranges.
|
|
@@ -141,11 +160,25 @@ export async function fetchPage(url) {
|
|
|
141
160
|
return impers ? viaImpers(impers, target) : viaFetch(target);
|
|
142
161
|
}
|
|
143
162
|
|
|
144
|
-
|
|
145
|
-
|
|
163
|
+
/**
|
|
164
|
+
* Follow redirects one hop at a time, validating each destination before the
|
|
165
|
+
* next request goes out.
|
|
166
|
+
*
|
|
167
|
+
* Both transports share this loop. They used to carry one each, which made the
|
|
168
|
+
* check that matters something a change could fix in one place and leave broken
|
|
169
|
+
* in the other, and made the guarantee testable only through a third party
|
|
170
|
+
* willing to 302 wherever it was told. Taking the request as a callback is what
|
|
171
|
+
* lets the hop check be proven against a transport that never leaves the
|
|
172
|
+
* process.
|
|
173
|
+
* @param {(url: string) => Promise<any>} get - one request, redirects not followed
|
|
174
|
+
* @param {string} start
|
|
175
|
+
* @returns {Promise<{res: any, url: string}>} the first non-redirect response
|
|
176
|
+
*/
|
|
177
|
+
export async function followRedirects(get, start) {
|
|
178
|
+
let current = start;
|
|
146
179
|
for (let i = 0; ; i++) {
|
|
147
|
-
if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${
|
|
148
|
-
const res = await
|
|
180
|
+
if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${start}`);
|
|
181
|
+
const res = await get(current);
|
|
149
182
|
const status = res.status ?? res.statusCode ?? 0;
|
|
150
183
|
const location = res.headers.get('location');
|
|
151
184
|
if (status >= 300 && status < 400 && location) {
|
|
@@ -153,56 +186,40 @@ async function followImpersRedirects(impers, startUrl, impersonate) {
|
|
|
153
186
|
await assertSafeTarget(current);
|
|
154
187
|
continue;
|
|
155
188
|
}
|
|
156
|
-
return res;
|
|
189
|
+
return { res, url: current };
|
|
157
190
|
}
|
|
158
191
|
}
|
|
159
192
|
|
|
160
193
|
async function viaImpers(impers, target) {
|
|
161
194
|
// Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a
|
|
162
195
|
// blocked first attempt gets one cheap retry with a second identity.
|
|
196
|
+
const asking = (impersonate) => (url) => impers.get(url, { impersonate, allowRedirects: false });
|
|
163
197
|
let via = 'impers:chrome';
|
|
164
|
-
let res = await
|
|
198
|
+
let { res } = await followRedirects(asking('chrome'), target);
|
|
165
199
|
let status = res.status ?? res.statusCode ?? 0;
|
|
166
200
|
if (status >= 400) {
|
|
167
201
|
via = 'impers:firefox';
|
|
168
|
-
res = await
|
|
202
|
+
({ res } = await followRedirects(asking('firefox'), target));
|
|
169
203
|
status = res.status ?? res.statusCode ?? 0;
|
|
170
204
|
}
|
|
171
205
|
if (status >= 400) throw new Error(`fetch failed: ${status} for ${target}`);
|
|
206
|
+
assertReadableType(res.headers.get('content-type'));
|
|
172
207
|
const html = typeof res.text === 'function' ? await res.text() : String(res.text ?? res.body ?? '');
|
|
173
208
|
return { url: res.url ?? target, html, status, via };
|
|
174
209
|
}
|
|
175
210
|
|
|
176
211
|
async function viaFetch(target) {
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
|
186
|
-
'accept-language': 'en-US,en;q=0.9',
|
|
187
|
-
},
|
|
188
|
-
});
|
|
189
|
-
const location = res.headers.get('location');
|
|
190
|
-
if (res.status >= 300 && res.status < 400 && location) {
|
|
191
|
-
current = new URL(location, current).toString();
|
|
192
|
-
await assertSafeTarget(current);
|
|
193
|
-
continue;
|
|
194
|
-
}
|
|
195
|
-
break;
|
|
196
|
-
}
|
|
212
|
+
const { res, url: current } = await followRedirects((url) => fetch(url, {
|
|
213
|
+
redirect: 'manual',
|
|
214
|
+
headers: {
|
|
215
|
+
'user-agent': UA,
|
|
216
|
+
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
|
|
217
|
+
'accept-language': 'en-US,en;q=0.9',
|
|
218
|
+
},
|
|
219
|
+
}), target);
|
|
197
220
|
if (!res.ok) {
|
|
198
221
|
throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${current}`);
|
|
199
222
|
}
|
|
200
|
-
|
|
201
|
-
// The impers path never checked the type at all, so this is also what keeps
|
|
202
|
-
// the two transports rendering the same URL the same way.
|
|
203
|
-
const type = res.headers.get('content-type') ?? '';
|
|
204
|
-
if (type && !/html|xml|json/.test(type)) {
|
|
205
|
-
throw new Error(`not a page oc can read (${type.split(';')[0]}), it renders HTML, XML feeds, and JSON`);
|
|
206
|
-
}
|
|
223
|
+
assertReadableType(res.headers.get('content-type'));
|
|
207
224
|
return { url: res.url || current, html: await res.text(), status: res.status, via: 'fetch' };
|
|
208
225
|
}
|
package/src/render.js
CHANGED
|
@@ -24,7 +24,7 @@ const num = (v) => v.toLocaleString('en-US');
|
|
|
24
24
|
// saving is only collected when the agent would have paged at all, while the
|
|
25
25
|
// overspend is paid on every page that runs a little long, including the ones
|
|
26
26
|
// answered by their first few lines. Four caps that overspend near 1,500 tokens.
|
|
27
|
-
const FINISH = 4;
|
|
27
|
+
export const FINISH = 4;
|
|
28
28
|
|
|
29
29
|
/**
|
|
30
30
|
* Budget-aware compact view of a distilled page. `from` is a position in the
|
|
@@ -162,5 +162,28 @@ export function formatBlock(b, { full = false } = {}) {
|
|
|
162
162
|
}
|
|
163
163
|
}
|
|
164
164
|
|
|
165
|
-
|
|
166
|
-
|
|
165
|
+
// A cut inside a sentence makes the half that is shown untrustworthy. Asked
|
|
166
|
+
// for the first sentence of a page, an agent was given it in full, followed by
|
|
167
|
+
// a truncation marker, and spent a turn on `read` to find out whether the
|
|
168
|
+
// sentence carried on. Ending on the last sentence that finished inside the cap
|
|
169
|
+
// answers that in the view itself. The floor bounds what the courtesy costs: a
|
|
170
|
+
// block whose only sentence end is early keeps the plain cut instead of
|
|
171
|
+
// throwing away a third of the window.
|
|
172
|
+
const SENTENCE_END = /[.!?]["')\]]*(?=\s)/g;
|
|
173
|
+
const SENTENCE_FLOOR = 0.7;
|
|
174
|
+
|
|
175
|
+
const truncate = (s) => {
|
|
176
|
+
if (s.length <= TEXT_CAP) return s;
|
|
177
|
+
// A line is to code what a sentence is to prose, and a code block is the only
|
|
178
|
+
// text that keeps its newlines, so the same courtesy applies: cut where a
|
|
179
|
+
// line ended. A period in code ends nothing, which is why this returns
|
|
180
|
+
// instead of falling through to the sentence rule below.
|
|
181
|
+
const line = s.slice(0, TEXT_CAP).lastIndexOf('\n');
|
|
182
|
+
if (line >= TEXT_CAP * SENTENCE_FLOOR) return `${s.slice(0, line)} ... +${num(s.length - line)} chars`;
|
|
183
|
+
let cut = TEXT_CAP;
|
|
184
|
+
for (const m of s.slice(0, TEXT_CAP).matchAll(SENTENCE_END)) {
|
|
185
|
+
const end = (m.index ?? 0) + m[0].length;
|
|
186
|
+
if (end >= TEXT_CAP * SENTENCE_FLOOR) cut = end;
|
|
187
|
+
}
|
|
188
|
+
return `${s.slice(0, cut).trimEnd()} ... +${num(s.length - cut)} chars`;
|
|
189
|
+
};
|