@only-cli/oc 0.3.0-beta.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -59,7 +59,8 @@ No setup at all also works: `npx @only-cli/oc` runs without a global install, an
59
59
  ```
60
60
  oc open <url> fetch and render a page with numbered actions
61
61
  oc do <n> follow the numbered link [n], or read [n] if it is text
62
- oc find <query> where a string appears on the page already open
62
+ oc find <query> where a string appears on the page already open, or
63
+ the region itself when only one place matches
63
64
  oc read <n> full text of the region at [n], up to 2000 tokens
64
65
  oc next the next budget worth of the page already open
65
66
  oc raw [url] distilled markdown of the whole page
@@ -69,7 +70,7 @@ oc submit [n] submit a form (planned)
69
70
 
70
71
  Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTML), `--session <name>`, `--verbose`/`-v` (metrics on stderr, or export `OC_VERBOSE=1`).
71
72
 
72
- `oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
73
+ `oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. A result title on a search page is a link, so `oc do` on it opens the result rather than repeating the title. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page, and a `find` with a single match prints that region instead of the number to read it with. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
73
74
 
74
75
  ## Supported websites
75
76
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@only-cli/oc",
3
- "version": "0.3.0-beta.1",
3
+ "version": "0.3.0",
4
4
  "description": "Turn websites into a compact CLI so AI agents can browse without burning tokens.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -8,12 +8,13 @@ description: Token-efficient web browsing and web content extraction for AI agen
8
8
  Renders a web page as a compact, numbered terminal view instead of raw HTML. A typical page is under 500 tokens.
9
9
 
10
10
  ```
11
- npx --yes @only-cli/oc@0.2.0 open <url> compact view, numbered elements
12
- npx --yes @only-cli/oc@0.2.0 do <n> follow link [n], or read it if [n] is text
13
- npx --yes @only-cli/oc@0.2.0 find <query> lines where a string appears, with numbers
14
- npx --yes @only-cli/oc@0.2.0 next next ~500 tokens of the page already open
15
- npx --yes @only-cli/oc@0.2.0 read <n> full text of region [n]
16
- npx --yes @only-cli/oc@0.2.0 raw [url] whole page as markdown (--html for cleaned HTML)
11
+ npx --yes @only-cli/oc@0.3.0 open <url> compact view, numbered elements
12
+ npx --yes @only-cli/oc@0.3.0 do <n> follow link [n], or read it if [n] is text
13
+ npx --yes @only-cli/oc@0.3.0 find <query> where a string appears, or that place itself
14
+ when only one matches
15
+ npx --yes @only-cli/oc@0.3.0 next next ~500 tokens of the page already open
16
+ npx --yes @only-cli/oc@0.3.0 read <n> full text of region [n]
17
+ npx --yes @only-cli/oc@0.3.0 raw [url] whole page as markdown (--html for cleaned HTML)
17
18
  ```
18
19
 
19
20
  None of these except `open`/`do`/`raw <url>` fetch anything — they replay the page `open` already saved.
@@ -23,13 +24,14 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
23
24
  - Line 1 is the title, then main content (article/thread/results); nav/sidebar/footer follow after `--- rest of page ---`, still numbered.
24
25
  - `--- repeated controls hidden ---`: per-item chrome (save/report/reply) dropped as repetitive; `raw` keeps it.
25
26
  - `[n]` marks a link, button, input, heading, or a text block long enough to be cut.
26
- - `... +820 chars`: block was cut there; `read <n>` prints it whole.
27
+ - Code blocks arrive as the page wrote them, lines and indentation intact, so a command in one can be run as printed.
28
+ - `... +820 chars`: block was cut there; `read <n>` prints it whole. The cut lands on the end of a sentence, or of a line in code, so what is shown is never half of one.
27
29
  - `... 164 more blocks (~7,100 tokens)`: rest of page past budget — a cost estimate, not a fetch. Omitted when the page would finish only a little over budget; then it's printed whole instead.
28
30
  - `actions:` footer lists valid next commands.
29
31
 
30
32
  ## Going further, cheapest first
31
33
 
32
- - `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit.
34
+ - `find <query>` — every place a string appears, one line + number each. Matches as a phrase (case-insensitive), falling back to separate words; reports how many matches didn't fit. When one place matches, or when the matches all fit, it prints them in full: no `read <n>` afterwards.
33
35
  - `read <n>` — one region in full: the block at `[n]` plus a little context, or the whole section for a heading.
34
36
  - `next` — continues the same page from where the budget stopped.
35
37
  - `raw` — everything, ~10x the cost. Use only when you need the whole page, not to hunt for a link's URL (use `do` for that).
@@ -42,7 +44,7 @@ None of these except `open`/`do`/`raw <url>` fetch anything — they replay the
42
44
  - `[6-9] 4 similar links` markers still work despite the collapsed text.
43
45
  - Search result links resolve to the destination, not the tracking redirect.
44
46
  - `do` on an input/button reports that instead (typing/submitting not yet supported).
45
- - `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow.
47
+ - `do` on a heading/text block prints the read instead of refusing, since there's nothing to follow. A heading that is itself a link, which is what a search result title is, opens instead.
46
48
  - `--session <name>` keeps separate page state, for working on two sites at once.
47
49
 
48
50
  ## Flags
package/src/act.js CHANGED
@@ -9,7 +9,7 @@
9
9
  */
10
10
 
11
11
  import { DEFAULT_SESSION, handleFor, handleNumbers, loadSession, saveSession } from './session.js';
12
- import { estimateTokens, formatBlock, render } from './render.js';
12
+ import { FINISH, estimateTokens, formatBlock, render } from './render.js';
13
13
 
14
14
  export class NotImplemented extends Error {
15
15
  constructor(command) {
@@ -63,7 +63,11 @@ export function activate(n, { session = DEFAULT_SESSION } = {}) {
63
63
  const range = nums.length ? `1-${Math.max(...nums)}` : 'none';
64
64
  throw new Error(`no [${n}] on ${state.url} (handles ${range}), run 'oc open <url>' again to renumber`);
65
65
  }
66
- if (handle.type === 'text' || handle.type === 'heading') {
66
+ // A heading can be a link: on a search results page the result title is one,
67
+ // and opening it is what `do` was asked for. Reading the title back instead
68
+ // cost a turn, and then another to find the number that does navigate, so a
69
+ // heading that has an href falls through to the link below.
70
+ if ((handle.type === 'text' || handle.type === 'heading') && !handle.href) {
67
71
  // There is nothing to follow, but the agent asked to see what is at [n],
68
72
  // and that is what read prints. Refusing would spend a whole turn to name
69
73
  // the command that should have run, and a turn costs more than the page.
@@ -208,14 +212,41 @@ export function find(query, { session = DEFAULT_SESSION, budget = 500 } = {}) {
208
212
  return `no match for "${query}"${tried} in ${blocks.length} blocks on ${state.url}, try fewer words or 'oc raw' for the full text`;
209
213
  }
210
214
 
211
- const lines = [`${hits.length} ${hits.length === 1 ? 'match' : 'matches'} for "${query}"${loose ? ', matching the words separately' : ''}`];
215
+ const separately = loose ? ', matching the words separately' : '';
216
+
217
+ // One match is not an index, it is the answer. Naming the number and
218
+ // stopping spends a turn to say where to look, and the agent's next command
219
+ // is always the `read` that looks, so find does the reading. Measured on an
220
+ // AWS CLI reference page, `find "Example 7"` printed a 24 token heading and
221
+ // the example it names was in the block after it.
222
+ if (hits.length === 1 && hits[0].n != null) {
223
+ const only = hits[0];
224
+ const follow = only.type === 'link' ? 'do <n> | ' : '';
225
+ return [
226
+ `1 match for "${query}"${separately}, region [${only.n}]`,
227
+ read(only.n, { session, budget: budget * FINISH }),
228
+ `actions: ${follow}read <n> | next | raw`,
229
+ ].join('\n');
230
+ }
231
+
232
+ const lines = [`${hits.length} ${hits.length === 1 ? 'match' : 'matches'} for "${query}"${separately}`];
233
+ // A snippet is short because many matches have to share one screen. When
234
+ // every match would fit whole inside the allowance a page already gets for
235
+ // finishing (render's FINISH), showing them whole makes this command the
236
+ // answer instead of an index into a `read <n>` that has to run next. The
237
+ // trade is the same one FINISH documents, and it is not close: a few
238
+ // hundred tokens against a turn.
239
+ const label = (hit, text) => `[${hit.n ?? '?'}] ${text}`;
240
+ const whole = hits.reduce((n, h) => n + estimateTokens(label(h, h.text)) + 1, estimateTokens(lines[0]));
241
+ const full = whole <= budget * FINISH;
242
+ const cap = full ? budget * FINISH : budget;
212
243
  let spent = estimateTokens(lines[0]);
213
244
  let shown = 0;
214
245
  let hasLinks = false;
215
246
  for (const hit of hits) {
216
- const line = `[${hit.n ?? '?'}] ${hit.snippet}`;
247
+ const line = label(hit, full ? hit.text : hit.snippet);
217
248
  const cost = estimateTokens(line) + 1;
218
- if (spent + cost > budget && shown) break;
249
+ if (spent + cost > cap && shown) break;
219
250
  spent += cost;
220
251
  shown++;
221
252
  if (hit.type === 'link') hasLinks = true;
@@ -251,8 +282,12 @@ function search(blocks, terms) {
251
282
  if (out.length && out.at(-1).n === n) continue;
252
283
  const start = Math.max(0, Math.min(...found) - BEFORE);
253
284
  const end = Math.min(block.text.length, start + SNIPPET);
254
- const snippet = `${start > 0 ? '... ' : ''}${block.text.slice(start, end)}${end < block.text.length ? ' ...' : ''}`;
255
- out.push({ n, type: block.type, snippet });
285
+ // One line per match is the promise the list makes, and a code block is the
286
+ // one kind of block that carries lines of its own. They survive where they
287
+ // are read rather than indexed: the whole-match mode above, and `read`.
288
+ const window = block.text.slice(start, end).replace(/\n/g, ' ');
289
+ const snippet = `${start > 0 ? '... ' : ''}${window}${end < block.text.length ? ' ...' : ''}`;
290
+ out.push({ n, type: block.type, snippet, text: block.text });
256
291
  }
257
292
  return out;
258
293
  }
package/src/cli.js CHANGED
@@ -11,7 +11,8 @@ const HELP = `only-cli: the web as a compact terminal, built for AI agents.
11
11
  usage: oc <command> [args] [flags]
12
12
 
13
13
  open <url> fetch and render a page with numbered actions
14
- find <query> where a string appears on the page already open
14
+ find <query> where a string appears on the page already open, or
15
+ the region itself when only one place matches
15
16
  next the next budget worth of the page already open
16
17
  read <n> full text of the region at [n], up to 2000 tokens
17
18
  raw [url] distilled markdown of the whole page
@@ -36,7 +37,8 @@ flags:
36
37
 
37
38
  'oc open' remembers the page it printed, so 'oc do 3' follows link [3] without
38
39
  you ever handling its URL, and 'oc next' or 'oc read 12' picks up what the
39
- budget left behind without fetching it again. State lives in ~/.only-cli
40
+ budget left behind without fetching it again. 'oc do' on a search result title
41
+ opens the result, because that title is a link. State lives in ~/.only-cli
40
42
  (override with OC_HOME).`;
41
43
 
42
44
  // A rendered page has to be remembered or its [3] means nothing to the next
package/src/distill.js CHANGED
@@ -7,7 +7,7 @@ import TurndownService from 'turndown';
7
7
  * @property {string} text
8
8
  * @property {number} [n] - action handle
9
9
  * @property {number} [level] - heading level 1..6
10
- * @property {string} [href] - links only
10
+ * @property {string} [href] - links, and a heading that is one
11
11
  * @property {string} [name] - inputs only
12
12
  *
13
13
  * @typedef {Object} Page
@@ -32,6 +32,32 @@ const DROP = new Set([
32
32
  // them, so the search below refuses to descend into them.
33
33
  const FURNITURE = new Set(['nav', 'header', 'footer', 'aside']);
34
34
 
35
+ // Controls a page puts inside its own code samples, and the selector that
36
+ // finds one. Node's API docs give every code block a copy button, a module
37
+ // toggle and a language label, so the text of the sample ends in
38
+ // `javascriptcopy` unless the toolbar holding them is left out of it.
39
+ const CODE_CONTROLS = new Set(['button', 'input', 'select', 'textarea', 'label']);
40
+ const CONTROL = [...CODE_CONTROLS].join(', ');
41
+
42
+ // Lines in a code block are kept, where every other kind of text has its
43
+ // whitespace collapsed. Joining them would put `// comment` in front of the
44
+ // statements that followed it, so a sample an agent could run would arrive
45
+ // commented out, and the shape of the wreck is invisible on one line. Blank
46
+ // runs and the indentation the whole block shares are the parts that carry no
47
+ // meaning, so those go.
48
+ const codeText = (raw) => {
49
+ const lines = raw.replace(/\r\n?/g, '\n').replace(/[^\S\n]+$/gm, '').split('\n');
50
+ while (lines.length && !lines[0].trim()) lines.shift();
51
+ while (lines.length && !lines[lines.length - 1].trim()) lines.pop();
52
+ const indent = lines
53
+ .filter((l) => l.trim())
54
+ .reduce((least, l) => Math.min(least, l.length - l.trimStart().length), Infinity);
55
+ return lines
56
+ .map((l) => (Number.isFinite(indent) ? l.slice(indent) : l).trimEnd())
57
+ .join('\n')
58
+ .replace(/\n{3,}/g, '\n\n');
59
+ };
60
+
35
61
  // Elements that end a line on the page and so must end one here. Without this
36
62
  // the text of six separate posts merges into a single block, because nothing
37
63
  // between them survives distillation to keep them apart.
@@ -82,6 +108,29 @@ const clean = (s) => s.replace(/\s+/g, ' ').trim();
82
108
  const asHTML = (text, url = '', opts = {}) =>
83
109
  jsonToHTML(text, url, opts) ?? youtubeToHTML(text) ?? transcriptToHTML(text) ?? feedToHTML(text) ?? text;
84
110
 
111
+ /**
112
+ * The link a heading is, if it is one. A search engine puts the result title
113
+ * in an anchor inside an <h2>, so a heading can be the most followable thing
114
+ * on the page, and taking only its text threw that away.
115
+ *
116
+ * The heading has to BE the link, not merely contain one: exactly one anchor,
117
+ * labelled with the whole heading. Documentation fails that test on purpose.
118
+ * Every heading in the Rust book and every one on an AWS CLI reference page
119
+ * carries a permalink to its own id, so following those would refetch the
120
+ * page the agent is already reading, which is worse than the reading it
121
+ * already gets. A bare fragment is never a destination.
122
+ * @param {any} node - the heading element
123
+ * @param {string} text - its cleaned text
124
+ * @returns {string|null}
125
+ */
126
+ function headingHref(node, text) {
127
+ const anchors = node.querySelectorAll('a[href]');
128
+ if (anchors.length !== 1) return null;
129
+ const href = anchors[0].getAttribute('href') ?? '';
130
+ if (!href || href.startsWith('#')) return null;
131
+ return clean(anchors[0].textContent) === text ? href : null;
132
+ }
133
+
85
134
  /**
86
135
  * Reduce raw HTML to an interaction tree: readable text plus numbered
87
136
  * elements, in document order. The walk is deterministic and numbering is a
@@ -105,6 +154,45 @@ export function distill(html, url = '') {
105
154
  /** Subtree already emitted, skipped when the rest of the page is walked. */
106
155
  let done = null;
107
156
 
157
+ /**
158
+ * The text of a code subtree, read as one string.
159
+ *
160
+ * A syntax highlighter gives every token its own element, so `s3://bucket/`
161
+ * reaches the walk as `s3`, `:`, `//`, `bucket`, `/`, and the rule that puts
162
+ * fragments back together only glues the ones that share a parent. The rest
163
+ * are space-joined, which turned an AWS example into
164
+ * `aws s3 cp s3 : // bucket / -- recursive`, a command an agent cannot run.
165
+ * Reading the subtree whole is what fixes that, and it discards nothing: over
166
+ * 172 pre elements on the AWS CLI reference, the Rust book, the Node API docs
167
+ * and the Python library docs, 159 are split this way and not one of them
168
+ * contains a link.
169
+ *
170
+ * textContent would be enough if pages put only code in their code blocks.
171
+ * A control inside one is chrome, and so is the block-level element holding
172
+ * it: that is how the toolbar's `javascript` label leaves with the copy
173
+ * button it sits beside. The test stays on block-level wrappers because a
174
+ * highlighter's own elements are inline, so a stray control can never take a
175
+ * line of code out with it.
176
+ * @param {any} node
177
+ * @returns {string}
178
+ */
179
+ const verbatim = (node) => {
180
+ let out = '';
181
+ const gather = (n) => {
182
+ if (n.nodeType === 3) {
183
+ out += n.textContent ?? '';
184
+ return;
185
+ }
186
+ if (n.nodeType !== 1) return;
187
+ const tag = n.localName;
188
+ if (DROP.has(tag) || CODE_CONTROLS.has(tag) || hidden(n)) return;
189
+ if (n !== node && BLOCKY.has(tag) && n.querySelector(CONTROL)) return;
190
+ for (const child of n.childNodes) gather(child);
191
+ };
192
+ gather(node);
193
+ return out;
194
+ };
195
+
108
196
  const walk = (node) => {
109
197
  if (node.nodeType === 3) {
110
198
  const raw = node.textContent ?? '';
@@ -126,7 +214,10 @@ export function distill(html, url = '') {
126
214
 
127
215
  if (/^h[1-6]$/.test(tag)) {
128
216
  const text = clean(node.textContent);
129
- if (text) blocks.push({ type: 'heading', level: Number(tag[1]), text });
217
+ if (text) {
218
+ const href = headingHref(node, text);
219
+ blocks.push({ type: 'heading', level: Number(tag[1]), text, ...(href ? { href } : {}) });
220
+ }
130
221
  return;
131
222
  }
132
223
  if (tag === 'a' && node.getAttribute('href')) {
@@ -157,6 +248,21 @@ export function distill(html, url = '') {
157
248
  if (text) blocks.push({ type: 'button', text });
158
249
  return;
159
250
  }
251
+ if (tag === 'pre' || tag === 'code') {
252
+ const raw = verbatim(node);
253
+ const text = tag === 'pre' ? codeText(raw) : clean(raw);
254
+ // A pre is its own line, which is what BLOCKY gave it before this branch
255
+ // started claiming it first. Inline code belongs to the sentence around
256
+ // it, so it carries the parent and the edge spacing a text node would and
257
+ // merges back into that sentence the same way.
258
+ const own = tag === 'pre';
259
+ if (own) blocks.push({ type: 'break' });
260
+ if (text) {
261
+ blocks.push({ type: 'text', text, host: node.parentNode, pre: /^\s/.test(raw), post: /\s$/.test(raw) });
262
+ }
263
+ if (own) blocks.push({ type: 'break' });
264
+ return;
265
+ }
160
266
  if (BLOCKY.has(tag)) {
161
267
  blocks.push({ type: 'break' });
162
268
  for (const child of node.childNodes) walk(child);
@@ -513,6 +619,15 @@ const TITLE_KEYS = [
513
619
  'full_name', 'summary', 'question', 'message',
514
620
  ];
515
621
 
622
+ // Keys an API is likely to put its list of results under. A response using one
623
+ // of these is a list whatever else it carries, so the name settles it before
624
+ // shape does: a sideloaded `included` array can outnumber the `items` the
625
+ // request was for without being what the request was for.
626
+ const CONTAINER_KEYS = [
627
+ 'items', 'data', 'results', 'hits', 'records', 'rows', 'entries',
628
+ 'nodes', 'edges', 'docs', 'list', 'children', 'values',
629
+ ];
630
+
516
631
  // Keys holding the item's own page. A URL under any other name is still found,
517
632
  // by looking at values rather than names, but these win when several qualify.
518
633
  const LINK_KEYS = ['link', 'url', 'html_url', 'web_url', 'permalink', 'href'];
@@ -540,6 +655,14 @@ const NESTED_PENALTY = 0.25;
540
655
  // How many names the footer lists when it says which fields it left out.
541
656
  const DROPPED_LISTED = 5;
542
657
 
658
+ // Characters of markup a field may carry before the compact view stops
659
+ // treating it as a document and starts treating it as somewhere to go. A
660
+ // question body arrives well under this and is worth rendering in place, links
661
+ // and code and all. A package readme arrives at four figures and distils into
662
+ // more blocks than the resource it is attached to has fields, which buries the
663
+ // resource the response was fetched for. `oc raw` renders either one in full.
664
+ const BODY_CAP = 4000;
665
+
543
666
  // Seconds and milliseconds since the epoch, bounded either side so an ordinary
544
667
  // count (a score, a byte size) is never mistaken for a date.
545
668
  const EPOCH_S = [1e9, 4e9];
@@ -572,6 +695,10 @@ const decodeEntities = (s) =>
572
695
  const isPlain = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
573
696
  const isURL = (v) => typeof v === 'string' && /^https?:\/\/\S+$/.test(v);
574
697
  const looksHTML = (v) => typeof v === 'string' && /<\/?(p|div|pre|code|br|ul|ol|li|h[1-6]|blockquote|table|img|a|em|strong)\b[^>]*>/i.test(v);
698
+ // Markup as one line of prose, for a body the compact view is pointing at
699
+ // rather than rendering. The distiller is what reads markup properly; this
700
+ // only has to make a line an agent can tell one body from another by.
701
+ const stripTags = (v) => clean(decodeEntities(String(v).replace(/<[^>]*>/g, ' ')));
575
702
 
576
703
  /**
577
704
  * One level of flattening, so `owner: {display_name}` becomes an
@@ -662,32 +789,40 @@ function chooseFields(rows, skip) {
662
789
  }
663
790
 
664
791
  /**
665
- * Pick the array the response is actually about: the root when it is one,
666
- * otherwise the longest array of objects at the top level, which is where
667
- * `items`, `data`, `results`, and `hits` all live. Everything beside it is
668
- * metadata about the request rather than content.
792
+ * Pick what the response is actually about: the root when it is an array or a
793
+ * single resource, otherwise the array of objects at the top level that holds
794
+ * the results. Everything beside it is metadata about the request rather than
795
+ * content.
669
796
  * @param {any} data
670
797
  * @returns {{items: any[], meta: Record<string, any>}}
671
798
  */
672
799
  function mainArray(data) {
673
800
  if (Array.isArray(data)) return { items: data, meta: {} };
674
- let key = '';
675
- /** @type {any[] | null} */
676
- let items = null;
801
+ /** @type {Map<string, any[]>} */
802
+ const arrays = new Map();
677
803
  for (const [k, v] of Object.entries(data)) {
678
804
  if (!Array.isArray(v) || !v.length) continue;
679
805
  if (!v.some(isPlain)) continue;
680
- if (!items || v.length > items.length) {
681
- items = v;
682
- key = k;
806
+ arrays.set(k, v);
807
+ }
808
+ let key = CONTAINER_KEYS.find((k) => arrays.has(k)) ?? '';
809
+ // A root carrying its own name is the resource, and an array hanging off it
810
+ // describes that resource rather than being the subject in its place. Taking
811
+ // the longest array regardless titled the npm registry's package endpoint
812
+ // after its two maintainers and demoted the package to the metadata line,
813
+ // where a 9KB readme then cost more than the rest of the page put together.
814
+ const named = TITLE_KEYS.some((k) => typeof data[k] === 'string' && data[k].trim() !== '');
815
+ if (!key && !named) {
816
+ for (const [k, v] of arrays) {
817
+ if (!key || v.length > (arrays.get(key)?.length ?? 0)) key = k;
683
818
  }
684
819
  }
685
- // A response with no array is a single resource, which renders as one item
686
- // rather than as a special case.
687
- if (!items) return { items: [data], meta: {} };
820
+ // A response with no results array of its own is a single resource, which
821
+ // renders as one item rather than as a special case.
822
+ if (!key) return { items: [data], meta: {} };
688
823
  const meta = { ...data };
689
824
  delete meta[key];
690
- return { items, meta };
825
+ return { items: arrays.get(key) ?? [data], meta };
691
826
  }
692
827
 
693
828
  /**
@@ -784,9 +919,16 @@ export function jsonToHTML(text, url = '', { full = false } = {}) {
784
919
  if (kept && !kept.has(key) && !looksHTML(value)) continue;
785
920
  // A field carrying HTML is a page in itself: `filter=withbody` on the
786
921
  // Stack Exchange API puts a whole question in one. It goes through the
787
- // distiller like any other markup instead of into a cell.
922
+ // distiller like any other markup instead of into a cell, unless it is
923
+ // longer than the compact view can afford, in which case it becomes one
924
+ // numbered line rather than a dozen blocks that bury the item it hangs
925
+ // off. `oc read <n>` opens it at a budget that fits it, `oc raw` always.
788
926
  if (looksHTML(value)) {
789
- bodies.push(String(value));
927
+ if (full || String(value).length <= BODY_CAP) {
928
+ bodies.push(String(value));
929
+ continue;
930
+ }
931
+ long.push(`<p>${escHTML(`${key}: ${stripTags(value)}`)}</p>`);
790
932
  continue;
791
933
  }
792
934
  const rendered = row.get(key);
@@ -824,12 +966,21 @@ export function jsonToHTML(text, url = '', { full = false } = {}) {
824
966
  }
825
967
 
826
968
  const metaBits = [];
969
+ const metaLong = [];
827
970
  for (const [key, value] of Object.entries(meta)) {
828
971
  if (value === null || typeof value === 'object') continue;
829
972
  const rendered = renderValue(key, value);
830
- if (rendered) metaBits.push(`${key}=${rendered}`);
973
+ if (!rendered) continue;
974
+ // A summary line has to stay a line. One long scalar at the root, a
975
+ // package readme or an endpoint description, would otherwise spend the
976
+ // whole page budget here, so it becomes a block of its own instead. The
977
+ // block is numbered, so `oc read <n>` opens it when it fits that budget
978
+ // and `oc raw` has it whatever its size.
979
+ if (rendered.length > TEXT_CAP) metaLong.push(`<p>${escHTML(`${key}: ${rendered}`)}</p>`);
980
+ else metaBits.push(`${key}=${rendered}`);
831
981
  }
832
982
  if (metaBits.length) parts.push(`<p>response: ${escHTML(metaBits.join(', '))}</p>`);
983
+ parts.push(...metaLong);
833
984
 
834
985
  const count = `${items.length} ${items.length === 1 ? 'item' : 'items'}`;
835
986
  return `<html><head><title>${escHTML(jsonTitle(url, count))}</title></head><body>\n${parts.join('\n')}\n</body></html>`;
package/src/fetch.js CHANGED
@@ -24,6 +24,25 @@ const loadImpers = () => {
24
24
  const BLOCKED_MESSAGE = 'blocked: private or internal URL';
25
25
  const MAX_REDIRECTS = 20;
26
26
 
27
+ // What oc can turn into text: any text/* type, plus the application/* types
28
+ // that are really text (json, xml, and the +json / +xml families a feed or an
29
+ // API answers with). A PNG matches none of these, and rendering one produces
30
+ // pages of mojibake an agent then pays for, so it is refused by name instead.
31
+ const READABLE_TYPE = /^\s*(?:text\/|application\/(?:json|xml|javascript|x-ndjson|[\w.+-]*\+(?:json|xml)))/i;
32
+
33
+ /**
34
+ * Refuse a response oc cannot read as text. Both transports call this: the
35
+ * gate has to live on whichever client got the page, or the same URL renders
36
+ * as an error through fetch and as binary noise through impers.
37
+ * @param {string | null | undefined} type - the content-type header
38
+ */
39
+ export function assertReadableType(type) {
40
+ // No header at all is not a refusal: plenty of small servers omit it, and
41
+ // the distiller handles whatever comes back.
42
+ if (!type || READABLE_TYPE.test(type)) return;
43
+ throw new Error(`not a page oc can read (${type.split(';')[0].trim()}), it renders HTML, XML feeds, JSON, and plain text`);
44
+ }
45
+
27
46
  // IPv4 ranges with no business receiving a server-initiated fetch: loopback,
28
47
  // link-local, the three RFC 1918 private blocks, carrier-grade NAT, the
29
48
  // unspecified/broadcast addresses, and the documentation/benchmark ranges.
@@ -141,11 +160,25 @@ export async function fetchPage(url) {
141
160
  return impers ? viaImpers(impers, target) : viaFetch(target);
142
161
  }
143
162
 
144
- async function followImpersRedirects(impers, startUrl, impersonate) {
145
- let current = startUrl;
163
+ /**
164
+ * Follow redirects one hop at a time, validating each destination before the
165
+ * next request goes out.
166
+ *
167
+ * Both transports share this loop. They used to carry one each, which made the
168
+ * check that matters something a change could fix in one place and leave broken
169
+ * in the other, and made the guarantee testable only through a third party
170
+ * willing to 302 wherever it was told. Taking the request as a callback is what
171
+ * lets the hop check be proven against a transport that never leaves the
172
+ * process.
173
+ * @param {(url: string) => Promise<any>} get - one request, redirects not followed
174
+ * @param {string} start
175
+ * @returns {Promise<{res: any, url: string}>} the first non-redirect response
176
+ */
177
+ export async function followRedirects(get, start) {
178
+ let current = start;
146
179
  for (let i = 0; ; i++) {
147
- if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${startUrl}`);
148
- const res = await impers.get(current, { impersonate, allowRedirects: false });
180
+ if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${start}`);
181
+ const res = await get(current);
149
182
  const status = res.status ?? res.statusCode ?? 0;
150
183
  const location = res.headers.get('location');
151
184
  if (status >= 300 && status < 400 && location) {
@@ -153,56 +186,40 @@ async function followImpersRedirects(impers, startUrl, impersonate) {
153
186
  await assertSafeTarget(current);
154
187
  continue;
155
188
  }
156
- return res;
189
+ return { res, url: current };
157
190
  }
158
191
  }
159
192
 
160
193
  async function viaImpers(impers, target) {
161
194
  // Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a
162
195
  // blocked first attempt gets one cheap retry with a second identity.
196
+ const asking = (impersonate) => (url) => impers.get(url, { impersonate, allowRedirects: false });
163
197
  let via = 'impers:chrome';
164
- let res = await followImpersRedirects(impers, target, 'chrome');
198
+ let { res } = await followRedirects(asking('chrome'), target);
165
199
  let status = res.status ?? res.statusCode ?? 0;
166
200
  if (status >= 400) {
167
201
  via = 'impers:firefox';
168
- res = await followImpersRedirects(impers, target, 'firefox');
202
+ ({ res } = await followRedirects(asking('firefox'), target));
169
203
  status = res.status ?? res.statusCode ?? 0;
170
204
  }
171
205
  if (status >= 400) throw new Error(`fetch failed: ${status} for ${target}`);
206
+ assertReadableType(res.headers.get('content-type'));
172
207
  const html = typeof res.text === 'function' ? await res.text() : String(res.text ?? res.body ?? '');
173
208
  return { url: res.url ?? target, html, status, via };
174
209
  }
175
210
 
176
211
  async function viaFetch(target) {
177
- let current = target;
178
- let res;
179
- for (let i = 0; ; i++) {
180
- if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${target}`);
181
- res = await fetch(current, {
182
- redirect: 'manual',
183
- headers: {
184
- 'user-agent': UA,
185
- accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
186
- 'accept-language': 'en-US,en;q=0.9',
187
- },
188
- });
189
- const location = res.headers.get('location');
190
- if (res.status >= 300 && res.status < 400 && location) {
191
- current = new URL(location, current).toString();
192
- await assertSafeTarget(current);
193
- continue;
194
- }
195
- break;
196
- }
212
+ const { res, url: current } = await followRedirects((url) => fetch(url, {
213
+ redirect: 'manual',
214
+ headers: {
215
+ 'user-agent': UA,
216
+ accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
217
+ 'accept-language': 'en-US,en;q=0.9',
218
+ },
219
+ }), target);
197
220
  if (!res.ok) {
198
221
  throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${current}`);
199
222
  }
200
- // JSON is a page here too: an API answer distills into one article per item.
201
- // The impers path never checked the type at all, so this is also what keeps
202
- // the two transports rendering the same URL the same way.
203
- const type = res.headers.get('content-type') ?? '';
204
- if (type && !/html|xml|json/.test(type)) {
205
- throw new Error(`not a page oc can read (${type.split(';')[0]}), it renders HTML, XML feeds, and JSON`);
206
- }
223
+ assertReadableType(res.headers.get('content-type'));
207
224
  return { url: res.url || current, html: await res.text(), status: res.status, via: 'fetch' };
208
225
  }
package/src/render.js CHANGED
@@ -24,7 +24,7 @@ const num = (v) => v.toLocaleString('en-US');
24
24
  // saving is only collected when the agent would have paged at all, while the
25
25
  // overspend is paid on every page that runs a little long, including the ones
26
26
  // answered by their first few lines. Four caps that overspend near 1,500 tokens.
27
- const FINISH = 4;
27
+ export const FINISH = 4;
28
28
 
29
29
  /**
30
30
  * Budget-aware compact view of a distilled page. `from` is a position in the
@@ -162,5 +162,28 @@ export function formatBlock(b, { full = false } = {}) {
162
162
  }
163
163
  }
164
164
 
165
- const truncate = (s) =>
166
- s.length > TEXT_CAP ? `${s.slice(0, TEXT_CAP)} ... +${num(s.length - TEXT_CAP)} chars` : s;
165
+ // A cut inside a sentence makes the half that is shown untrustworthy. Asked
166
+ // for the first sentence of a page, an agent was given it in full, followed by
167
+ // a truncation marker, and spent a turn on `read` to find out whether the
168
+ // sentence carried on. Ending on the last sentence that finished inside the cap
169
+ // answers that in the view itself. The floor bounds what the courtesy costs: a
170
+ // block whose only sentence end is early keeps the plain cut instead of
171
+ // throwing away a third of the window.
172
+ const SENTENCE_END = /[.!?]["')\]]*(?=\s)/g;
173
+ const SENTENCE_FLOOR = 0.7;
174
+
175
+ const truncate = (s) => {
176
+ if (s.length <= TEXT_CAP) return s;
177
+ // A line is to code what a sentence is to prose, and a code block is the only
178
+ // text that keeps its newlines, so the same courtesy applies: cut where a
179
+ // line ended. A period in code ends nothing, which is why this returns
180
+ // instead of falling through to the sentence rule below.
181
+ const line = s.slice(0, TEXT_CAP).lastIndexOf('\n');
182
+ if (line >= TEXT_CAP * SENTENCE_FLOOR) return `${s.slice(0, line)} ... +${num(s.length - line)} chars`;
183
+ let cut = TEXT_CAP;
184
+ for (const m of s.slice(0, TEXT_CAP).matchAll(SENTENCE_END)) {
185
+ const end = (m.index ?? 0) + m[0].length;
186
+ if (end >= TEXT_CAP * SENTENCE_FLOOR) cut = end;
187
+ }
188
+ return `${s.slice(0, cut).trimEnd()} ... +${num(s.length - cut)} chars`;
189
+ };