pi-quiver 4.1.0 → 4.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +24 -1
- package/dist/bin/pi-quiver.js +578 -0
- package/extensions/fetch.ts +6 -570
- package/extensions/session-name.ts +13 -1
- package/lib/fetch-core.ts +589 -0
- package/package.json +12 -5
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,14 @@ Published to npm as `pi-quiver` (`pi install npm:pi-quiver`). Pushing a
|
|
|
8
8
|
via OIDC trusted publishing. The release helper at
|
|
9
9
|
`.agents/skills/release/scripts/release.sh` cuts the tag; CI publishes.
|
|
10
10
|
|
|
11
|
+
## v4.2.0 - 2026-08-20
|
|
12
|
+
|
|
13
|
+
- fetch: data plane extracted to `lib/fetch-core.ts`; new `pi-quiver fetch` CLI (esbuild-built `dist/` bin) with full parameter parity; Claude Code skill + plugin marketplace (`quiver:fetch` via `npx -y pi-quiver@latest`). pi tool behavior unchanged.
|
|
14
|
+
|
|
15
|
+
## v4.1.1 - 2026-08-17
|
|
16
|
+
|
|
17
|
+
- **`session-name`: fix silent auto-naming failure on GitHub Copilot business/enterprise accounts.** Those credentials pin requests to an account-specific endpoint (`auth.baseUrl` from `getApiKeyAndHeaders`); the naming call ignored it and hit the catalog's individual endpoint, failing with `421 Misdirected Request` and leaving sessions unnamed. The naming request now mirrors pi's own request path and prefers the credential's endpoint.
|
|
18
|
+
|
|
11
19
|
## v4.1.0 - 2026-08-14
|
|
12
20
|
|
|
13
21
|
- **`session-name`: configurable naming policy** (#5). New `sessionAutoName` keys: `rules` (house conventions appended to the naming prompt, later rules win), `deny` (literal case-insensitive phrases stripped from every name, loose interior whitespace so `"acme corp"` also catches `AcmeCorp`), and `revisitFirstTurn`/`revisitEveryTurns` (re-derive the name once those round-trip counts are crossed; default `0` = off, each revisit is one short LLM call). Machine-generated names are replaced when stale; human-set names only get a non-blocking suggestion notification. Name provenance and cadence persist across resume. Revisits run detached and only after the agent has fully settled, so automated multi-turn runs (chains, workflows) are never renamed or delayed mid-flight.
|
package/README.md
CHANGED
|
@@ -62,7 +62,7 @@ A 300 KB changelog page never touches your context window - you get a preview an
|
|
|
62
62
|
|
|
63
63
|
| Extension | Tool | What it does |
|
|
64
64
|
| --- | --- | --- |
|
|
65
|
-
| `extensions/fetch.ts` | `fetch` | Retrieve URLs over HTTP(S). HTML -> Markdown (Readability extraction, Turndown conversion). Binary saved untouched to a temp file. GitHub issue/PR/repo/actions-run URLs auto-route through `gh` (falls back to HTTP). Same size gate as `doc_to_md`. |
|
|
65
|
+
| `extensions/fetch.ts` | `fetch` | Retrieve URLs over HTTP(S). HTML -> Markdown (Readability extraction, Turndown conversion). Binary saved untouched to a temp file. GitHub issue/PR/repo/actions-run URLs auto-route through `gh` (falls back to HTTP). Same size gate as `doc_to_md`. Behavior lives in `lib/fetch-core.ts`; also exposed as the `pi-quiver fetch` CLI (see [Claude Code support](#claude-code-support)). |
|
|
66
66
|
| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/PPTX to Markdown. High-fidelity via `pymupdf4llm` (run through `uv`); degraded pure-JS fallback (`unpdf`) when `uv`/Python is unavailable or conversion times out. DOCX/PPTX convert via LibreOffice first. |
|
|
67
67
|
| `extensions/session-name.ts` | `/session-name` | Manual + opt-in automatic session naming, naming rules and deny list, long-session revisits, and Ghostty tab rename. OFF by default. |
|
|
68
68
|
| `extensions/sword-header.ts` | `/builtin-header` | Themed ASCII startup header replacing pi's default logo. OFF by default. |
|
|
@@ -205,6 +205,29 @@ Operational notes:
|
|
|
205
205
|
- **A watchdog abort that the provider ignores escalates after a fixed 10s.** Any post-abort stream event re-arms that deadline (bytes prove only that the connection was alive at that instant), so a stream that emits a straggler and then wedges still escalates 10s after its last event. This reduces the hang; it cannot force the provider to stop, and undici's timeouts remain the final backstop.
|
|
206
206
|
- **Headless runs report on stderr.** In `print`/`json` mode pi binds a no-op UI, so watchdog notices go out via `console.warn`. Nothing is ever written to stdout, which `json` mode uses for its protocol. In TUI and RPC the notices render as main-window notifications, not the bottom status line.
|
|
207
207
|
|
|
208
|
+
## Claude Code support
|
|
209
|
+
|
|
210
|
+
`fetch`'s core (`lib/fetch-core.ts`) is also published as a CLI, so Claude Code can use the same routing, size gate, and spill behavior as pi's native tool - without pi ever seeing Claude-only files.
|
|
211
|
+
|
|
212
|
+
**Exposed:** the `quiver` plugin, served from this repo's `.claude-plugin/marketplace.json`, with one skill: `fetch` (invoked as `quiver:fetch` / `/quiver:fetch`). The skill runs `npx -y pi-quiver@latest fetch <url> [flags]` via Bash - full parameter parity with the pi tool (`--method`, `--header`, `--body`, `--raw`, `--timeout-ms`), same GitHub `gh` routing, same size gate, same binary-to-temp-file handling. See [doc/fetch.md](doc/fetch.md#claude-code-cli-pi-quiver-fetch) for exit codes and flags.
|
|
213
|
+
|
|
214
|
+
**Not exposed:** pi extensions, `doc_to_md`, and everything else in this package - the marketplace allowlists only `./skills/fetch`, and the npm tarball never ships `skills/` or `.claude-plugin/` (pi's own `files` allowlist excludes them, and pi's explicit `pi.extensions` manifest makes them invisible to pi's convention-directory auto-discovery either way).
|
|
215
|
+
|
|
216
|
+
Add the marketplace and enable the plugin in `.claude/settings.json`:
|
|
217
|
+
|
|
218
|
+
```json
|
|
219
|
+
{
|
|
220
|
+
"extraKnownMarketplaces": {
|
|
221
|
+
"pi-quiver": { "source": { "source": "github", "repo": "jjuraszek/pi-quiver" } }
|
|
222
|
+
},
|
|
223
|
+
"enabledPlugins": { "quiver@pi-quiver": true }
|
|
224
|
+
}
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Activates on folder trust.
|
|
228
|
+
|
|
229
|
+
**Release sequencing:** the skill goes live only with (or after) the npm release that ships the `pi-quiver` bin - until that tag is on npm, `npx -y pi-quiver@latest fetch` resolves a bin-less package and fails.
|
|
230
|
+
|
|
208
231
|
## Development
|
|
209
232
|
|
|
210
233
|
Deps are peers (`@earendil-works/*`, `@sinclair/typebox`) plus the bundled
|
|
@@ -0,0 +1,578 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
// bin/pi-quiver.ts
|
|
4
|
+
import { realpathSync } from "node:fs";
|
|
5
|
+
import { fileURLToPath } from "node:url";
|
|
6
|
+
import { resolve } from "node:path";
|
|
7
|
+
|
|
8
|
+
// lib/fetch-core.ts
|
|
9
|
+
import { mkdirSync, writeFileSync, createWriteStream } from "node:fs";
|
|
10
|
+
import { rm } from "node:fs/promises";
|
|
11
|
+
import { tmpdir } from "node:os";
|
|
12
|
+
import { join } from "node:path";
|
|
13
|
+
import { createHash } from "node:crypto";
|
|
14
|
+
import { execFile } from "node:child_process";
|
|
15
|
+
import { promisify } from "node:util";
|
|
16
|
+
import { JSDOM } from "jsdom";
|
|
17
|
+
import { Readability } from "@mozilla/readability";
|
|
18
|
+
import TurndownService from "turndown";
|
|
19
|
+
import { gfm } from "turndown-plugin-gfm";
|
|
20
|
+
function formatSize(bytes) {
|
|
21
|
+
if (bytes < 1024) {
|
|
22
|
+
return `${bytes}B`;
|
|
23
|
+
} else if (bytes < 1024 * 1024) {
|
|
24
|
+
return `${(bytes / 1024).toFixed(1)}KB`;
|
|
25
|
+
} else {
|
|
26
|
+
return `${(bytes / (1024 * 1024)).toFixed(1)}MB`;
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
var RESERVED_OWNERS = /* @__PURE__ */ new Set([
|
|
30
|
+
"orgs",
|
|
31
|
+
"users",
|
|
32
|
+
"sponsors",
|
|
33
|
+
"topics",
|
|
34
|
+
"marketplace",
|
|
35
|
+
"apps",
|
|
36
|
+
"collections",
|
|
37
|
+
"stars",
|
|
38
|
+
"settings",
|
|
39
|
+
"notifications",
|
|
40
|
+
"codespaces",
|
|
41
|
+
"features",
|
|
42
|
+
"trending",
|
|
43
|
+
"security",
|
|
44
|
+
"customer-stories"
|
|
45
|
+
]);
|
|
46
|
+
var GH_NAME = /^[A-Za-z0-9._-]+$/;
|
|
47
|
+
function classifyGitHubTarget(url) {
|
|
48
|
+
const host = url.hostname.toLowerCase();
|
|
49
|
+
if (host !== "github.com" && host !== "www.github.com") return null;
|
|
50
|
+
const segs = url.pathname.split("/").filter((s) => s.length > 0);
|
|
51
|
+
if (segs.length < 2) return null;
|
|
52
|
+
const [owner, repo] = segs;
|
|
53
|
+
if (!GH_NAME.test(owner) || !GH_NAME.test(repo)) return null;
|
|
54
|
+
if (RESERVED_OWNERS.has(owner.toLowerCase())) return null;
|
|
55
|
+
if (segs.length === 4 && segs[2] === "issues" && /^\d+$/.test(segs[3])) {
|
|
56
|
+
return { kind: "issue", url: `https://github.com/${owner}/${repo}/issues/${segs[3]}` };
|
|
57
|
+
}
|
|
58
|
+
if (segs.length === 4 && segs[2] === "pull" && /^\d+$/.test(segs[3])) {
|
|
59
|
+
return { kind: "pr", url: `https://github.com/${owner}/${repo}/pull/${segs[3]}` };
|
|
60
|
+
}
|
|
61
|
+
if (segs.length === 5 && segs[2] === "actions" && segs[3] === "runs" && /^\d+$/.test(segs[4])) {
|
|
62
|
+
return { kind: "run", slug: `${owner}/${repo}`, runId: segs[4], url: `https://github.com/${owner}/${repo}/actions/runs/${segs[4]}` };
|
|
63
|
+
}
|
|
64
|
+
if (segs.length === 2) {
|
|
65
|
+
return { kind: "repo", slug: `${owner}/${repo}` };
|
|
66
|
+
}
|
|
67
|
+
return null;
|
|
68
|
+
}
|
|
69
|
+
function buildGhArgs(target) {
|
|
70
|
+
if (target.kind === "issue") return ["issue", "view", target.url, "--comments"];
|
|
71
|
+
if (target.kind === "pr") return ["pr", "view", target.url, "--comments"];
|
|
72
|
+
if (target.kind === "run") return ["run", "view", target.runId, "--repo", target.slug];
|
|
73
|
+
return ["repo", "view", target.slug];
|
|
74
|
+
}
|
|
75
|
+
var GH_MAX_BUFFER = 1e7;
|
|
76
|
+
var execFileAsync = promisify(execFile);
|
|
77
|
+
var runGh = async (args, timeoutMs, signal) => {
|
|
78
|
+
try {
|
|
79
|
+
const { stdout } = await execFileAsync("gh", args, {
|
|
80
|
+
timeout: timeoutMs,
|
|
81
|
+
signal,
|
|
82
|
+
maxBuffer: GH_MAX_BUFFER,
|
|
83
|
+
encoding: "utf8"
|
|
84
|
+
});
|
|
85
|
+
if (!stdout.trim()) return { ok: false };
|
|
86
|
+
return { ok: true, stdout };
|
|
87
|
+
} catch {
|
|
88
|
+
return { ok: false };
|
|
89
|
+
}
|
|
90
|
+
};
|
|
91
|
+
function planGhRouting(params, url) {
|
|
92
|
+
if (params.raw) return null;
|
|
93
|
+
if ((params.method ?? "GET") !== "GET") return null;
|
|
94
|
+
if (params.body) return null;
|
|
95
|
+
if (params.headers && Object.keys(params.headers).length > 0) return null;
|
|
96
|
+
return classifyGitHubTarget(url);
|
|
97
|
+
}
|
|
98
|
+
function ghCommandLabel(target) {
|
|
99
|
+
if (target.kind === "issue") return "issue view --comments";
|
|
100
|
+
if (target.kind === "pr") return "pr view --comments";
|
|
101
|
+
if (target.kind === "run") return "run view";
|
|
102
|
+
return "repo view";
|
|
103
|
+
}
|
|
104
|
+
function ghSourceLine(target, ref) {
|
|
105
|
+
if (target.kind === "issue") return `gh issue view ${ref} --comments`;
|
|
106
|
+
if (target.kind === "pr") return `gh pr view ${ref} --comments`;
|
|
107
|
+
if (target.kind === "run") return `gh run view ${target.runId} --repo ${target.slug}`;
|
|
108
|
+
return `gh repo view ${ref}`;
|
|
109
|
+
}
|
|
110
|
+
function renderGhResult(target, stdout) {
|
|
111
|
+
const body = stdout.trimEnd();
|
|
112
|
+
const ref = target.kind === "repo" ? target.slug : target.url;
|
|
113
|
+
const { spill, bytes, lines } = applyGate(body);
|
|
114
|
+
const baseDetails = {
|
|
115
|
+
url: ref,
|
|
116
|
+
bytes,
|
|
117
|
+
lines,
|
|
118
|
+
category: "markdown",
|
|
119
|
+
via: "gh",
|
|
120
|
+
ghCommand: ghCommandLabel(target)
|
|
121
|
+
};
|
|
122
|
+
const source = `Source: ${ghSourceLine(target, ref)}`;
|
|
123
|
+
if (!spill) {
|
|
124
|
+
return {
|
|
125
|
+
output: [source, "", body].join("\n"),
|
|
126
|
+
details: { ...baseDetails, spilled: false }
|
|
127
|
+
};
|
|
128
|
+
}
|
|
129
|
+
const spillUrl = target.kind === "repo" ? `https://github.com/${target.slug}` : target.url;
|
|
130
|
+
const file = spillToFile(spillUrl, body, "md");
|
|
131
|
+
return {
|
|
132
|
+
output: [
|
|
133
|
+
source,
|
|
134
|
+
`Body: ${formatSize(bytes)} across ${lines} lines \u2014 written to file (too large to inline)`,
|
|
135
|
+
`Saved-To: ${file}`,
|
|
136
|
+
"",
|
|
137
|
+
"Read slices of this file with the read tool (offset/limit) or grep it; do not read the whole file unless you must. Markdown is grep-able by heading (^#).",
|
|
138
|
+
"",
|
|
139
|
+
`----- preview (first ${PREVIEW_LINES} lines) -----`,
|
|
140
|
+
buildPreview(body)
|
|
141
|
+
].join("\n"),
|
|
142
|
+
details: { ...baseDetails, spilled: true, file }
|
|
143
|
+
};
|
|
144
|
+
}
|
|
145
|
+
async function executeGhRouting(params, url, signal, runner = runGh) {
|
|
146
|
+
const target = planGhRouting(params, url);
|
|
147
|
+
if (!target) return null;
|
|
148
|
+
const gh = await runner(buildGhArgs(target), params.timeoutMs ?? DEFAULT_TIMEOUT_MS, signal);
|
|
149
|
+
if (!gh.ok) return null;
|
|
150
|
+
return renderGhResult(target, gh.stdout);
|
|
151
|
+
}
|
|
152
|
+
var PARSABLE_MAX_BYTES = 1e6;
|
|
153
|
+
var BINARY_MAX_BYTES = 5e7;
|
|
154
|
+
var SNIFF_MAX_BYTES = 64e3;
|
|
155
|
+
var DEFAULT_TIMEOUT_MS = 2e4;
|
|
156
|
+
var INLINE_MAX_BYTES = 32e3;
|
|
157
|
+
var INLINE_MAX_LINES = 1e3;
|
|
158
|
+
var PREVIEW_LINES = 60;
|
|
159
|
+
var PREVIEW_MAX_BYTES = 4e3;
|
|
160
|
+
var FIREFOX_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.7; rv:135.0) Gecko/20100101 Firefox/135.0";
|
|
161
|
+
var DEFAULT_ACCEPT = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
|
|
162
|
+
function parseCharset(contentType) {
|
|
163
|
+
const m = /charset\s*=\s*"?([^";\s]+)"?/i.exec(contentType);
|
|
164
|
+
return (m?.[1] ?? "utf-8").trim().toLowerCase();
|
|
165
|
+
}
|
|
166
|
+
function decodeBuffer(buf, charset) {
|
|
167
|
+
try {
|
|
168
|
+
return new TextDecoder(charset, { fatal: false }).decode(buf);
|
|
169
|
+
} catch {
|
|
170
|
+
return new TextDecoder("utf-8", { fatal: false }).decode(buf);
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
function buildPreview(body) {
|
|
174
|
+
let preview = body.split("\n").slice(0, PREVIEW_LINES).join("\n");
|
|
175
|
+
if (preview.length > PREVIEW_MAX_BYTES) {
|
|
176
|
+
preview = `${preview.slice(0, PREVIEW_MAX_BYTES)}
|
|
177
|
+
\u2026[preview truncated]`;
|
|
178
|
+
}
|
|
179
|
+
return preview;
|
|
180
|
+
}
|
|
181
|
+
var turndownService = new TurndownService({
|
|
182
|
+
headingStyle: "atx",
|
|
183
|
+
codeBlockStyle: "fenced",
|
|
184
|
+
bulletListMarker: "-"
|
|
185
|
+
});
|
|
186
|
+
turndownService.use(gfm);
|
|
187
|
+
function mimeType(contentType) {
|
|
188
|
+
return contentType.split(";")[0].trim().toLowerCase();
|
|
189
|
+
}
|
|
190
|
+
var TEXT_ALLOWLIST = [
|
|
191
|
+
/^text\//,
|
|
192
|
+
/^application\/(json|xml|xhtml\+xml|javascript)$/,
|
|
193
|
+
/\+json$/,
|
|
194
|
+
/\+xml$/
|
|
195
|
+
];
|
|
196
|
+
var KNOWN_BINARY = [
|
|
197
|
+
/^audio\//,
|
|
198
|
+
/^video\//,
|
|
199
|
+
/^font\//,
|
|
200
|
+
/^application\/(pdf|zip|gzip|x-tar|x-7z-compressed|x-rar-compressed|wasm)$/
|
|
201
|
+
];
|
|
202
|
+
function categorize(contentType, sniff, raw) {
|
|
203
|
+
const mime = mimeType(contentType);
|
|
204
|
+
if (/^image\//.test(mime)) return "binary";
|
|
205
|
+
const isText = TEXT_ALLOWLIST.some((re) => re.test(mime));
|
|
206
|
+
const isBinary = KNOWN_BINARY.some((re) => re.test(mime));
|
|
207
|
+
if (!isText && !isBinary) return sniff.includes(0) ? "binary" : "text";
|
|
208
|
+
if (isBinary && !isText) return "binary";
|
|
209
|
+
if (sniff.includes(0)) return "binary";
|
|
210
|
+
if (raw) return "text";
|
|
211
|
+
if (mime === "text/html" || mime === "application/xhtml+xml") return "markdown";
|
|
212
|
+
if (mime === "application/json" || /\+json$/.test(mime)) return "json";
|
|
213
|
+
return "text";
|
|
214
|
+
}
|
|
215
|
+
function htmlToMarkdown(html, url) {
|
|
216
|
+
let doc;
|
|
217
|
+
try {
|
|
218
|
+
doc = new JSDOM(html, { url }).window.document;
|
|
219
|
+
} catch {
|
|
220
|
+
return null;
|
|
221
|
+
}
|
|
222
|
+
let article = null;
|
|
223
|
+
try {
|
|
224
|
+
article = new Readability(doc).parse();
|
|
225
|
+
} catch {
|
|
226
|
+
return null;
|
|
227
|
+
}
|
|
228
|
+
if (!article?.content) return null;
|
|
229
|
+
let md;
|
|
230
|
+
try {
|
|
231
|
+
md = turndownService.turndown(article.content).trim();
|
|
232
|
+
} catch {
|
|
233
|
+
return null;
|
|
234
|
+
}
|
|
235
|
+
if (!md) return null;
|
|
236
|
+
if (article.title) md = `# ${article.title}
|
|
237
|
+
|
|
238
|
+
${md}`;
|
|
239
|
+
return md;
|
|
240
|
+
}
|
|
241
|
+
function prettyJson(text) {
|
|
242
|
+
try {
|
|
243
|
+
return JSON.stringify(JSON.parse(text), null, 2);
|
|
244
|
+
} catch {
|
|
245
|
+
return text;
|
|
246
|
+
}
|
|
247
|
+
}
|
|
248
|
+
function applyGate(body) {
|
|
249
|
+
const bytes = Buffer.byteLength(body, "utf8");
|
|
250
|
+
const lines = body.length ? body.split("\n").length : 0;
|
|
251
|
+
const spill = body.length > 0 && (bytes > INLINE_MAX_BYTES || lines > INLINE_MAX_LINES);
|
|
252
|
+
return { spill, bytes, lines };
|
|
253
|
+
}
|
|
254
|
+
function tempFilePath(url, ext) {
|
|
255
|
+
const dir = join(tmpdir(), "pi-fetch");
|
|
256
|
+
mkdirSync(dir, { recursive: true });
|
|
257
|
+
let host = "page";
|
|
258
|
+
try {
|
|
259
|
+
host = new URL(url).hostname.replace(/[^a-z0-9.-]/gi, "_") || "page";
|
|
260
|
+
} catch {
|
|
261
|
+
}
|
|
262
|
+
const hash = createHash("sha1").update(url).digest("hex").slice(0, 8);
|
|
263
|
+
const stamp = (/* @__PURE__ */ new Date()).toISOString().replace(/[:.]/g, "-");
|
|
264
|
+
return join(dir, `${stamp}-${host}-${hash}.${ext}`);
|
|
265
|
+
}
|
|
266
|
+
function spillToFile(url, body, ext) {
|
|
267
|
+
const file = tempFilePath(url, ext);
|
|
268
|
+
writeFileSync(file, body, "utf8");
|
|
269
|
+
return file;
|
|
270
|
+
}
|
|
271
|
+
function textExtension(category, contentType) {
|
|
272
|
+
if (category === "markdown") return "md";
|
|
273
|
+
if (category === "json") return "json";
|
|
274
|
+
return mimeType(contentType).includes("xml") ? "xml" : "txt";
|
|
275
|
+
}
|
|
276
|
+
var BINARY_EXT = {
|
|
277
|
+
"application/pdf": "pdf",
|
|
278
|
+
"application/zip": "zip",
|
|
279
|
+
"application/vnd.openxmlformats-officedocument.wordprocessingml.document": "docx",
|
|
280
|
+
"application/vnd.openxmlformats-officedocument.presentationml.presentation": "pptx",
|
|
281
|
+
"application/gzip": "gz",
|
|
282
|
+
"image/png": "png",
|
|
283
|
+
"image/jpeg": "jpg",
|
|
284
|
+
"image/gif": "gif",
|
|
285
|
+
"image/webp": "webp",
|
|
286
|
+
"image/svg+xml": "svg"
|
|
287
|
+
};
|
|
288
|
+
function binaryExtension(contentType) {
|
|
289
|
+
const mime = mimeType(contentType);
|
|
290
|
+
if (BINARY_EXT[mime]) return BINARY_EXT[mime];
|
|
291
|
+
const sub = (mime.split("/")[1] ?? "").replace(/^x-/, "").replace(/[^a-z0-9]+/g, "").slice(0, 8);
|
|
292
|
+
return sub || "bin";
|
|
293
|
+
}
|
|
294
|
+
function writeChunk(stream, b) {
|
|
295
|
+
return new Promise((resolve2, reject) => {
|
|
296
|
+
stream.write(b, (err) => err ? reject(err) : resolve2());
|
|
297
|
+
});
|
|
298
|
+
}
|
|
299
|
+
async function pumpToFile(stream, reader, prefix, exhausted) {
|
|
300
|
+
let bytes = 0;
|
|
301
|
+
let truncated = false;
|
|
302
|
+
let head = prefix;
|
|
303
|
+
if (head.length > BINARY_MAX_BYTES) {
|
|
304
|
+
head = head.subarray(0, BINARY_MAX_BYTES);
|
|
305
|
+
truncated = true;
|
|
306
|
+
}
|
|
307
|
+
await writeChunk(stream, head);
|
|
308
|
+
bytes += head.length;
|
|
309
|
+
while (!exhausted && !truncated) {
|
|
310
|
+
const { done, value } = await reader.read();
|
|
311
|
+
if (done) break;
|
|
312
|
+
let chunk = Buffer.from(value);
|
|
313
|
+
if (bytes + chunk.length > BINARY_MAX_BYTES) {
|
|
314
|
+
chunk = chunk.subarray(0, BINARY_MAX_BYTES - bytes);
|
|
315
|
+
truncated = true;
|
|
316
|
+
}
|
|
317
|
+
await writeChunk(stream, chunk);
|
|
318
|
+
bytes += chunk.length;
|
|
319
|
+
}
|
|
320
|
+
await new Promise((resolve2, reject) => stream.end((err) => err ? reject(err) : resolve2()));
|
|
321
|
+
return { bytes, truncated };
|
|
322
|
+
}
|
|
323
|
+
async function collectBody(res, contentType, raw) {
|
|
324
|
+
const reader = res.body.getReader();
|
|
325
|
+
const prefixParts = [];
|
|
326
|
+
let prefixLen = 0;
|
|
327
|
+
let exhausted = false;
|
|
328
|
+
while (prefixLen < SNIFF_MAX_BYTES) {
|
|
329
|
+
const { done, value } = await reader.read();
|
|
330
|
+
if (done) {
|
|
331
|
+
exhausted = true;
|
|
332
|
+
break;
|
|
333
|
+
}
|
|
334
|
+
const chunk = Buffer.from(value);
|
|
335
|
+
prefixParts.push(chunk);
|
|
336
|
+
prefixLen += chunk.length;
|
|
337
|
+
}
|
|
338
|
+
const prefix = Buffer.concat(prefixParts);
|
|
339
|
+
const category = categorize(contentType, prefix.subarray(0, SNIFF_MAX_BYTES), raw);
|
|
340
|
+
if (category === "binary") {
|
|
341
|
+
const file = tempFilePath(res.url, binaryExtension(contentType));
|
|
342
|
+
const stream = createWriteStream(file);
|
|
343
|
+
try {
|
|
344
|
+
const { bytes: bytes2, truncated: truncated2 } = await pumpToFile(stream, reader, prefix, exhausted);
|
|
345
|
+
if (truncated2) await reader.cancel().catch(() => {
|
|
346
|
+
});
|
|
347
|
+
return { category, file, bytes: bytes2, truncated: truncated2 };
|
|
348
|
+
} catch (err) {
|
|
349
|
+
stream.destroy();
|
|
350
|
+
await rm(file, { force: true });
|
|
351
|
+
await reader.cancel().catch(() => {
|
|
352
|
+
});
|
|
353
|
+
throw err;
|
|
354
|
+
}
|
|
355
|
+
}
|
|
356
|
+
const parts = [prefix];
|
|
357
|
+
let bytes = prefix.length;
|
|
358
|
+
let streamDone = exhausted;
|
|
359
|
+
while (!streamDone && bytes < PARSABLE_MAX_BYTES) {
|
|
360
|
+
const { done, value } = await reader.read();
|
|
361
|
+
if (done) {
|
|
362
|
+
streamDone = true;
|
|
363
|
+
break;
|
|
364
|
+
}
|
|
365
|
+
const chunk = Buffer.from(value);
|
|
366
|
+
parts.push(chunk);
|
|
367
|
+
bytes += chunk.length;
|
|
368
|
+
}
|
|
369
|
+
let buffer = Buffer.concat(parts);
|
|
370
|
+
if (buffer.length > PARSABLE_MAX_BYTES) {
|
|
371
|
+
buffer = buffer.subarray(0, PARSABLE_MAX_BYTES);
|
|
372
|
+
}
|
|
373
|
+
const truncated = !streamDone;
|
|
374
|
+
if (truncated) await reader.cancel().catch(() => {
|
|
375
|
+
});
|
|
376
|
+
return { category, buffer, bytes: buffer.length, truncated };
|
|
377
|
+
}
|
|
378
|
+
async function fetchUrl(opts) {
|
|
379
|
+
const url = new URL(opts.url);
|
|
380
|
+
if (url.protocol !== "http:" && url.protocol !== "https:") {
|
|
381
|
+
throw new Error(`Unsupported protocol: ${url.protocol}`);
|
|
382
|
+
}
|
|
383
|
+
const ghResult = await executeGhRouting(opts, url, opts.signal);
|
|
384
|
+
if (ghResult) return ghResult;
|
|
385
|
+
const headers = new Headers(opts.headers ?? {});
|
|
386
|
+
if (!headers.has("user-agent")) headers.set("user-agent", FIREFOX_UA);
|
|
387
|
+
if (!headers.has("accept")) headers.set("accept", DEFAULT_ACCEPT);
|
|
388
|
+
if (!headers.has("accept-language"))
|
|
389
|
+
headers.set("accept-language", "en-US,en;q=0.5");
|
|
390
|
+
const controller = new AbortController();
|
|
391
|
+
const onAbort = () => controller.abort();
|
|
392
|
+
opts.signal?.addEventListener("abort", onAbort);
|
|
393
|
+
const timer = setTimeout(
|
|
394
|
+
() => controller.abort(new Error("fetch timeout")),
|
|
395
|
+
opts.timeoutMs ?? DEFAULT_TIMEOUT_MS
|
|
396
|
+
);
|
|
397
|
+
try {
|
|
398
|
+
const res = await fetch(url, {
|
|
399
|
+
method: opts.method ?? "GET",
|
|
400
|
+
headers,
|
|
401
|
+
body: opts.body,
|
|
402
|
+
signal: controller.signal,
|
|
403
|
+
redirect: "follow"
|
|
404
|
+
});
|
|
405
|
+
const ct = res.headers.get("content-type") ?? "";
|
|
406
|
+
const charset = parseCharset(ct);
|
|
407
|
+
const header = [
|
|
408
|
+
`HTTP ${res.status} ${res.statusText}`,
|
|
409
|
+
`Content-Type: ${ct}`,
|
|
410
|
+
`Charset: ${charset}`
|
|
411
|
+
];
|
|
412
|
+
if (!res.body || (opts.method ?? "GET") === "HEAD") {
|
|
413
|
+
return {
|
|
414
|
+
output: [...header, "Length: 0 (no body)"].join("\n"),
|
|
415
|
+
details: { url: res.url, status: res.status, contentType: ct, charset, bytes: 0 }
|
|
416
|
+
};
|
|
417
|
+
}
|
|
418
|
+
const collected = await collectBody(res, ct, opts.raw ?? false);
|
|
419
|
+
const baseDetails = {
|
|
420
|
+
url: res.url,
|
|
421
|
+
status: res.status,
|
|
422
|
+
contentType: ct,
|
|
423
|
+
charset,
|
|
424
|
+
bytes: collected.bytes,
|
|
425
|
+
truncated: collected.truncated,
|
|
426
|
+
category: collected.category
|
|
427
|
+
};
|
|
428
|
+
if (collected.category === "binary") {
|
|
429
|
+
const note = collected.truncated ? " (truncated to 50MB)" : "";
|
|
430
|
+
return {
|
|
431
|
+
output: [
|
|
432
|
+
...header,
|
|
433
|
+
`Body: ${formatSize(collected.bytes)}${note} binary (${mimeType(ct) || "unknown"}) \u2014 saved untouched for processing`,
|
|
434
|
+
`Saved-To: ${collected.file}`,
|
|
435
|
+
"",
|
|
436
|
+
"Binary content is not decoded. Use the appropriate tool to process the file at the path above."
|
|
437
|
+
].join("\n"),
|
|
438
|
+
details: { ...baseDetails, spilled: true, file: collected.file }
|
|
439
|
+
};
|
|
440
|
+
}
|
|
441
|
+
const decoded = decodeBuffer(collected.buffer, charset);
|
|
442
|
+
let body;
|
|
443
|
+
let effectiveCategory = collected.category;
|
|
444
|
+
if (collected.category === "markdown") {
|
|
445
|
+
const md = htmlToMarkdown(decoded, res.url);
|
|
446
|
+
if (md !== null) {
|
|
447
|
+
body = md;
|
|
448
|
+
} else {
|
|
449
|
+
body = decoded;
|
|
450
|
+
effectiveCategory = "text";
|
|
451
|
+
}
|
|
452
|
+
} else if (collected.category === "json") {
|
|
453
|
+
body = prettyJson(decoded);
|
|
454
|
+
} else {
|
|
455
|
+
body = decoded;
|
|
456
|
+
}
|
|
457
|
+
baseDetails.category = effectiveCategory;
|
|
458
|
+
const truncNote = collected.truncated ? "\n[Note: source truncated at 1MB \u2014 content may be partial]" : "";
|
|
459
|
+
const lengthLine = `Length: ${collected.bytes}${collected.truncated ? " (truncated to 1MB)" : ""}`;
|
|
460
|
+
const { spill, bytes: bodyBytes, lines: lineCount } = applyGate(body);
|
|
461
|
+
baseDetails.lines = lineCount;
|
|
462
|
+
if (!spill) {
|
|
463
|
+
return {
|
|
464
|
+
output: [...header, lengthLine, "", body + truncNote].join("\n"),
|
|
465
|
+
details: { ...baseDetails, spilled: false }
|
|
466
|
+
};
|
|
467
|
+
}
|
|
468
|
+
const ext = textExtension(effectiveCategory, ct);
|
|
469
|
+
const file = spillToFile(res.url, body, ext);
|
|
470
|
+
const grepHint = effectiveCategory === "markdown" ? "Read slices of this file with the read tool (offset/limit) or grep it; do not read the whole file unless you must. Markdown is grep-able by heading (^#)." : "Read slices of this file with the read tool (offset/limit) or grep it; do not read the whole file unless you must.";
|
|
471
|
+
return {
|
|
472
|
+
output: [
|
|
473
|
+
...header,
|
|
474
|
+
lengthLine,
|
|
475
|
+
`Body: ${formatSize(bodyBytes)} across ${lineCount} lines \u2014 written to file (too large to inline)`,
|
|
476
|
+
`Saved-To: ${file}`,
|
|
477
|
+
...collected.truncated ? ["[Note: source truncated at 1MB \u2014 content may be partial]"] : [],
|
|
478
|
+
"",
|
|
479
|
+
grepHint,
|
|
480
|
+
"",
|
|
481
|
+
`----- preview (first ${PREVIEW_LINES} lines) -----`,
|
|
482
|
+
buildPreview(body)
|
|
483
|
+
].join("\n"),
|
|
484
|
+
details: { ...baseDetails, spilled: true, file }
|
|
485
|
+
};
|
|
486
|
+
} finally {
|
|
487
|
+
clearTimeout(timer);
|
|
488
|
+
opts.signal?.removeEventListener("abort", onAbort);
|
|
489
|
+
}
|
|
490
|
+
}
|
|
491
|
+
|
|
492
|
+
// bin/pi-quiver.ts
|
|
493
|
+
var USAGE = 'Usage: pi-quiver fetch <url> [--method GET|HEAD|POST] [--header "K: V"]... [--body <str>] [--raw] [--timeout-ms <n>]';
|
|
494
|
+
function parseCliArgs(argv) {
|
|
495
|
+
if (argv[0] !== "fetch") return { ok: false, error: `unknown command: ${argv[0] ?? "(none)"}` };
|
|
496
|
+
const rest = argv.slice(1);
|
|
497
|
+
let url;
|
|
498
|
+
let method;
|
|
499
|
+
let headers;
|
|
500
|
+
let body;
|
|
501
|
+
let raw;
|
|
502
|
+
let timeoutMs;
|
|
503
|
+
for (let i = 0; i < rest.length; i++) {
|
|
504
|
+
const arg = rest[i];
|
|
505
|
+
if (arg === "--raw") {
|
|
506
|
+
raw = true;
|
|
507
|
+
continue;
|
|
508
|
+
}
|
|
509
|
+
if (arg === "--method" || arg === "--header" || arg === "--body" || arg === "--timeout-ms") {
|
|
510
|
+
const value = rest[++i];
|
|
511
|
+
if (value === void 0) return { ok: false, error: `${arg} requires a value` };
|
|
512
|
+
if (arg === "--method") {
|
|
513
|
+
if (value !== "GET" && value !== "HEAD" && value !== "POST") {
|
|
514
|
+
return { ok: false, error: `invalid --method: ${value}` };
|
|
515
|
+
}
|
|
516
|
+
method = value;
|
|
517
|
+
} else if (arg === "--header") {
|
|
518
|
+
const sep = value.indexOf(": ");
|
|
519
|
+
if (sep <= 0) return { ok: false, error: `malformed --header (expected "Key: Value"): ${value}` };
|
|
520
|
+
headers ??= {};
|
|
521
|
+
headers[value.slice(0, sep)] = value.slice(sep + 2);
|
|
522
|
+
} else if (arg === "--body") {
|
|
523
|
+
body = value;
|
|
524
|
+
} else {
|
|
525
|
+
const n = Number(value);
|
|
526
|
+
if (!Number.isFinite(n) || n <= 0) return { ok: false, error: `invalid --timeout-ms: ${value}` };
|
|
527
|
+
timeoutMs = n;
|
|
528
|
+
}
|
|
529
|
+
continue;
|
|
530
|
+
}
|
|
531
|
+
if (arg.startsWith("--")) return { ok: false, error: `unknown flag: ${arg}` };
|
|
532
|
+
if (url !== void 0) return { ok: false, error: `unexpected argument: ${arg}` };
|
|
533
|
+
url = arg;
|
|
534
|
+
}
|
|
535
|
+
if (!url) return { ok: false, error: "missing <url>" };
|
|
536
|
+
const opts = { url };
|
|
537
|
+
if (method !== void 0) opts.method = method;
|
|
538
|
+
if (headers !== void 0) opts.headers = headers;
|
|
539
|
+
if (body !== void 0) opts.body = body;
|
|
540
|
+
if (raw !== void 0) opts.raw = raw;
|
|
541
|
+
if (timeoutMs !== void 0) opts.timeoutMs = timeoutMs;
|
|
542
|
+
return { ok: true, opts };
|
|
543
|
+
}
|
|
544
|
+
async function main() {
|
|
545
|
+
const parsed = parseCliArgs(process.argv.slice(2));
|
|
546
|
+
if (!parsed.ok) {
|
|
547
|
+
process.stderr.write(`${parsed.error}
|
|
548
|
+
${USAGE}
|
|
549
|
+
`);
|
|
550
|
+
return 2;
|
|
551
|
+
}
|
|
552
|
+
try {
|
|
553
|
+
const result = await fetchUrl(parsed.opts);
|
|
554
|
+
process.stdout.write(`${result.output}
|
|
555
|
+
`);
|
|
556
|
+
return 0;
|
|
557
|
+
} catch (err) {
|
|
558
|
+
process.stderr.write(`fetch failed: ${err instanceof Error ? err.message : String(err)}
|
|
559
|
+
`);
|
|
560
|
+
return 1;
|
|
561
|
+
}
|
|
562
|
+
}
|
|
563
|
+
function isMainEntry() {
|
|
564
|
+
if (!process.argv[1]) return false;
|
|
565
|
+
try {
|
|
566
|
+
return realpathSync(fileURLToPath(import.meta.url)) === realpathSync(resolve(process.argv[1]));
|
|
567
|
+
} catch {
|
|
568
|
+
return false;
|
|
569
|
+
}
|
|
570
|
+
}
|
|
571
|
+
if (isMainEntry()) {
|
|
572
|
+
main().then((code) => {
|
|
573
|
+
process.exitCode = code;
|
|
574
|
+
});
|
|
575
|
+
}
|
|
576
|
+
export {
|
|
577
|
+
parseCliArgs
|
|
578
|
+
};
|