@nailuogg/pi-find-packages 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,40 @@
1
+ name: publish
2
+
3
+ on:
4
+ push:
5
+ tags: ["v*"]
6
+
7
+ permissions:
8
+ contents: write # create the GitHub release
9
+ id-token: write # reserved for a future Trusted-Publishing (OIDC) switch
10
+
11
+ jobs:
12
+ publish:
13
+ runs-on: ubuntu-latest
14
+ steps:
15
+ - uses: actions/checkout@v4
16
+
17
+ - uses: actions/setup-node@v4
18
+ with:
19
+ node-version: 24
20
+ registry-url: https://registry.npmjs.org/
21
+ # reads NODE_AUTH_TOKEN from env to write ~/.npmrc
22
+
23
+ - name: Verify version matches tag
24
+ run: |
25
+ PKG_VERSION="${GITHUB_REF_NAME#v}"
26
+ TAG_VERSION="$(node -p "require('./package.json').version")"
27
+ if [ "${PKG_VERSION}" != "${TAG_VERSION}" ]; then
28
+ echo "::error::tag ${GITHUB_REF_NAME} != package.json version ${TAG_VERSION}"
29
+ exit 1
30
+ fi
31
+
32
+ - name: Publish
33
+ run: npm publish
34
+ env:
35
+ NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
36
+
37
+ - name: Create GitHub release
38
+ run: gh release create "${GITHUB_REF_NAME}" --verify-tag --generate-notes
39
+ env:
40
+ GH_TOKEN: ${{ github.token }}
@@ -0,0 +1,45 @@
1
+ name: update-data
2
+
3
+ on:
4
+ schedule:
5
+ - cron: "17 3 * * *" # daily, off the hour
6
+ workflow_dispatch:
7
+
8
+ permissions:
9
+ contents: write
10
+
11
+ jobs:
12
+ refresh:
13
+ runs-on: ubuntu-latest
14
+ steps:
15
+ - uses: actions/checkout@v4
16
+
17
+ - uses: actions/setup-node@v4
18
+ with:
19
+ node-version: 24
20
+
21
+ - name: Fetch latest catalog
22
+ run: node scripts/sync-catalog.mjs --out "${RUNNER_TEMP}/catalog"
23
+
24
+ - name: Skip if unchanged
25
+ id: diff
26
+ run: |
27
+ NEW="${RUNNER_TEMP}/catalog/catalog.jsonl.gz"
28
+ OLD="$(curl -fsSL --max-time 30 https://cdn.jsdelivr.net/gh/${GITHUB_REPOSITORY}@data/data/catalog.jsonl.gz 2>/dev/null | sha256sum | cut -d' ' -f1 || true)"
29
+ CUR="$(sha256sum "${NEW}" | cut -d' ' -f1)"
30
+ echo "changed=$([[ "${CUR}" != "${OLD}" ]] && echo true || echo false)" >> "$GITHUB_OUTPUT"
31
+
32
+ - name: Push to data branch
33
+ if: steps.diff.outputs.changed == 'true'
34
+ run: |
35
+ git config user.name "github-actions[bot]"
36
+ git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
37
+ git fetch origin data || true
38
+ git checkout -B data origin/data 2>/dev/null || git checkout -B data
39
+ mkdir -p data
40
+ cp "${RUNNER_TEMP}/catalog/catalog.jsonl" data/
41
+ cp "${RUNNER_TEMP}/catalog/catalog.jsonl.gz" data/
42
+ cp "${RUNNER_TEMP}/catalog/catalog.jsonl.gz.sha256" data/
43
+ git add data/
44
+ git commit -qm "chore(data): refresh catalog ($(date -u +%Y-%m-%d))"
45
+ git push origin data
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 nailuoGG
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,68 @@
1
+ # pi-find-packages
2
+
3
+ A [pi](https://pi.dev) extension providing a local catalog of the pi package ecosystem plus a `/find-packages` integration-review workflow.
4
+
5
+ ## Features
6
+
7
+ - **Offline catalog**: syncs the full `keywords:pi-package` corpus (~5k+ packages) from npm to local JSONL; search with jq/grep over descriptions, names, and keywords — no dependence on npm's keyword-matched search
8
+ - **`/find-packages <need>`**: retrieve candidates → read-only source analysis in a sandbox → comparison table and recommendation against criteria (feature overlap / peer compatibility / maintenance activity / supply-chain signals)
9
+ - **Cold start**: the package ships a bundled catalog snapshot (`data/catalog.jsonl.gz`); install and use immediately with zero network requests on first run
10
+ - **Docker isolation by default**: candidate repos are cloned and unpacked inside a credential-free, non-root container; isolation can be disabled explicitly (a risk warning is shown on every use — not recommended)
11
+
12
+ ## Install
13
+
14
+ ```bash
15
+ pi install npm:@nailuogg/pi-find-packages
16
+ pi install git:github.com/nailuoGG/pi-find-packages
17
+ pi install /path/to/pi-find-packages # local trial
18
+ ```
19
+
20
+ ## Data directory
21
+
22
+ `<PI_CODING_AGENT_DIR>/data/pi-find-packages/` (default `~/.pi/agent/data/pi-find-packages/`)
23
+
24
+ - `catalog.jsonl` — catalog data (one package per line: name/version/description/date/author/keywords/repo)
25
+ - `config.json` — `{"isolation": "docker" | "off", "semantic": "auto" | "on" | "off"}`
26
+ - `isolation` defaults to `docker` (sandboxed source analysis)
27
+ - `semantic` controls semantic search over lazily-cached READMEs (via [qmd](https://github.com/tobi/qmd)); defaults to `auto`: enabled automatically when the `qmd` binary is present, force with `"on"`, disable with `"off"`
28
+
29
+ ## Refreshing the catalog
30
+
31
+ Preferred, in order:
32
+
33
+ 1. `/find-packages update` — pulls the latest snapshot from the jsDelivr `data` branch
34
+ (`cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data`), falls back to the npmmirror
35
+ tarball of the latest published version. Checksum-verified, atomic replace.
36
+ 2. Full rebuild from the npm registry search API (~40 requests):
37
+
38
+ ```bash
39
+ node <package-dir>/scripts/sync-catalog.mjs # default data dir
40
+ node scripts/sync-catalog.mjs --out /tmp/catalog # custom output dir (used by CI)
41
+ ```
42
+
43
+ Once every ≥30 days is plenty — please avoid hammering the npm registry.
44
+
45
+ A GitHub Actions workflow (`update-data.yml`) refreshes the `data` branch daily when the
46
+ corpus changes, which is what the jsDelivr channel serves. The npmmirror channel tracks
47
+ the latest npm publish and therefore updates on release cadence.
48
+
49
+ ## Analysis sandbox
50
+
51
+ ```bash
52
+ docker build -t pi-find-packages-analysis -f docker/Dockerfile.analysis docker
53
+ ```
54
+
55
+ The image contains no pi and no credentials: it exists solely to clone/unpack/read third-party source. Never execute a candidate package's install scripts or build artifacts in any environment.
56
+
57
+ ## Releasing
58
+
59
+ Publishing is automated: push a tag `v<version>` matching `package.json`, and the
60
+ `publish.yml` workflow verifies the version, publishes `@nailuogg/pi-find-packages`
61
+ to npm, and creates a GitHub release. The `NPM_TOKEN` repository secret must hold a
62
+ granular npm access token scoped to the `@nailuogg` packages (npmjs.com → Access Keys).
63
+
64
+ ## Security boundaries
65
+
66
+ - An analysis report is **not** install authorization — whether to `pi install` is always the user's decision
67
+ - `isolation: off` injects a risk warning on every use; third-party package source may contain malicious logic
68
+ - Planned: update channel via GitHub tag + jsDelivr CDN with checksum verification
Binary file
@@ -0,0 +1,13 @@
1
+ FROM node:24-bookworm-slim
2
+
3
+ RUN apt-get update \
4
+ && apt-get install -y --no-install-recommends bash ca-certificates git ripgrep curl jq \
5
+ && rm -rf /var/lib/apt/lists/*
6
+
7
+ # No pi, no credentials: this image only clones/unpacks and reads third-party source.
8
+ RUN useradd -m -s /bin/bash analyst
9
+ RUN mkdir -p /analysis && chown analyst:analyst /analysis
10
+ WORKDIR /analysis
11
+ USER analyst
12
+
13
+ ENTRYPOINT ["bash"]
@@ -0,0 +1,190 @@
1
+ /**
2
+ * /find-packages — search the local pi-package catalog and kick off guided analysis.
3
+ *
4
+ * Data dir: <agentDir>/data/pi-find-packages/catalog.jsonl
5
+ * Cold start: extracts the bundled data/catalog.jsonl.gz on first run.
6
+ * Config (config.json in the data dir):
7
+ * isolation: "docker" (default) | "off" — source-analysis execution environment
8
+ * semantic: "auto" (default) | "on" | "off" — qmd-backed semantic search over cached READMEs
9
+ * Subcommand: /find-packages update — refresh the catalog from jsDelivr (data branch),
10
+ * then the npmmirror tarball of the latest published version; local sync script as last resort.
11
+ */
12
+ import { existsSync, mkdirSync, readFileSync, writeFileSync, renameSync } from "node:fs";
13
+ import { spawnSync } from "node:child_process";
14
+ import { gunzipSync } from "node:zlib";
15
+ import { createHash } from "node:crypto";
16
+ import { tmpdir } from "node:os";
17
+ import { join, dirname } from "node:path";
18
+ import { fileURLToPath } from "node:url";
19
+ import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
20
+
21
+ const JSDELIVR_GZ = "https://cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data/data/catalog.jsonl.gz";
22
+ const JSDELIVR_SHA = "https://cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data/data/catalog.jsonl.gz.sha256";
23
+ const NPMIRROR_TARBALL = "https://registry.npmmirror.com/@nailuogg%2Fpi-find-packages/latest";
24
+
25
+ export default function activate(pi: ExtensionAPI) {
26
+ // PI_CODING_AGENT_DIR is the official config-dir override (docs/environment-variables.md).
27
+ const agentDir = process.env.PI_CODING_AGENT_DIR || join(process.env.HOME ?? "", ".pi/agent");
28
+ const dataDir = join(agentDir, "data/pi-find-packages");
29
+ const catalog = join(dataDir, "catalog.jsonl");
30
+ const configFile = join(dataDir, "config.json");
31
+
32
+ // Cold start: extract bundled snapshot if catalog missing.
33
+ const pkgDir = dirname(dirname(fileURLToPath(import.meta.url))); // extensions/../ = package root
34
+ const bundled = join(pkgDir, "data/catalog.jsonl.gz");
35
+ if (!existsSync(catalog)) {
36
+ mkdirSync(dataDir, { recursive: true });
37
+ if (existsSync(bundled)) {
38
+ writeFileSync(catalog, gunzipSync(readFileSync(bundled)));
39
+ pi.appendEntry("find-packages", { event: "cold-start-extract", from: bundled });
40
+ }
41
+ }
42
+
43
+ function readConfig(): Record<string, unknown> {
44
+ try {
45
+ return JSON.parse(readFileSync(configFile, "utf8"));
46
+ } catch {
47
+ return {};
48
+ }
49
+ }
50
+
51
+ function isolation(): "docker" | "off" {
52
+ return readConfig().isolation === "off" ? "off" : "docker"; // secure default
53
+ }
54
+
55
+ // Semantic search via qmd: "auto" (default) = enabled iff qmd binary exists; "on" forces; "off" disables.
56
+ let qmdAvailable: boolean | undefined;
57
+ function semanticEnabled() {
58
+ const pref = readConfig().semantic;
59
+ if (pref === "off") return false;
60
+ if (qmdAvailable === undefined) {
61
+ try {
62
+ qmdAvailable = spawnSync("qmd", ["--version"], { timeout: 5000 }).status === 0;
63
+ } catch {
64
+ qmdAvailable = false;
65
+ }
66
+ }
67
+ return qmdAvailable;
68
+ }
69
+
70
+ async function fetchText(url: string, timeoutMs = 15000): Promise<string | undefined> {
71
+ try {
72
+ const res = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) });
73
+ if (!res.ok) return undefined;
74
+ return await res.text();
75
+ } catch {
76
+ return undefined;
77
+ }
78
+ }
79
+
80
+ async function fetchBuffer(url: string, timeoutMs = 30000): Promise<Buffer | undefined> {
81
+ try {
82
+ const res = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) });
83
+ if (!res.ok) return undefined;
84
+ return Buffer.from(await res.arrayBuffer());
85
+ } catch {
86
+ return undefined;
87
+ }
88
+ }
89
+
90
+ function sha256(buf: Buffer): string {
91
+ return createHash("sha256").update(buf).digest("hex");
92
+ }
93
+
94
+ function installCatalog(buf: Buffer, ctx): boolean {
95
+ const tmp = join(tmpdir(), `catalog.jsonl.${Date.now()}`);
96
+ try {
97
+ const jsonl = gunzipSync(buf);
98
+ JSON.parse(jsonl.toString("utf8").split("\n")[0]); // sanity: first line parses
99
+ writeFileSync(tmp, jsonl);
100
+ renameSync(tmp, catalog);
101
+ return true;
102
+ } catch {
103
+ try { spawnSync("rm", ["-f", tmp]); } catch {}
104
+ ctx.ui.notify("Downloaded catalog failed validation; keeping the existing file.", "error");
105
+ return false;
106
+ }
107
+ }
108
+
109
+ async function updateCatalog(ctx): Promise<void> {
110
+ // 1) jsDelivr data branch (freshest; updated by CI)
111
+ const gz = await fetchBuffer(JSDELIVR_GZ);
112
+ const expected = (await fetchText(JSDELIVR_SHA))?.trim();
113
+ if (gz && expected && sha256(gz) === expected.split(/\s+/)[0]) {
114
+ if (installCatalog(gz, ctx)) ctx.ui.notify("Catalog updated from jsDelivr (data branch).", "info");
115
+ return;
116
+ }
117
+ // 2) npmmirror tarball of the latest published version (freshness = publish cadence)
118
+ try {
119
+ const meta = await (await fetch(NPMIRROR_TARBALL, { signal: AbortSignal.timeout(15000) })).json();
120
+ const tarballUrl: string | undefined = meta?.dist?.tarball;
121
+ if (tarballUrl) {
122
+ const res = await fetch(tarballUrl, { signal: AbortSignal.timeout(60000) });
123
+ if (res.ok) {
124
+ const buf = Buffer.from(await res.arrayBuffer());
125
+ const proc = spawnSync("tar", ["-xzOf", "-", "package/data/catalog.jsonl.gz"], { input: buf });
126
+ if (proc.status === 0 && proc.stdout?.length) {
127
+ if (installCatalog(Buffer.from(proc.stdout), ctx)) {
128
+ ctx.ui.notify("Catalog updated from npmmirror (latest published version).", "info");
129
+ return;
130
+ }
131
+ }
132
+ }
133
+ }
134
+ } catch {}
135
+ // 3) fall back to the local sync script
136
+ ctx.ui.notify(
137
+ "Both CDNs unavailable or checksum mismatch. Rebuild locally with:\n"
138
+ + ` node ${join(pkgDir, "scripts/sync-catalog.mjs")}`,
139
+ "warning",
140
+ );
141
+ }
142
+
143
+ pi.registerCommand("find-packages", {
144
+ description: "Search local pi-package catalog for packages matching a need; `update` refreshes the catalog",
145
+ handler: async (args, ctx) => {
146
+ const query = (args ?? "").trim();
147
+ if (query === "update") {
148
+ await updateCatalog(ctx);
149
+ return;
150
+ }
151
+ if (!query) {
152
+ ctx.ui.notify("Usage: /find-packages <need> — e.g. /find-packages control sessions remotely; or /find-packages update", "warning");
153
+ return;
154
+ }
155
+ if (!existsSync(catalog)) {
156
+ ctx.ui.notify(`catalog.jsonl not found. Run: /find-packages update — or: node ${join(pkgDir, "scripts/sync-catalog.mjs")}`, "error");
157
+ return;
158
+ }
159
+ const iso = isolation();
160
+ const riskNote =
161
+ iso === "off"
162
+ ? [
163
+ "",
164
+ "⚠️ **Isolation disabled (isolation: off)**: candidate source will be cloned/unpacked on the host.",
165
+ "Third-party packages may contain malicious install scripts or build logic; host-side analysis is not sandboxed.",
166
+ "This mode is not recommended. Delete data/pi-find-packages/config.json or set isolation:\"docker\" to restore the default.",
167
+ ].join("\n")
168
+ : "";
169
+ const prompt = [
170
+ `Search the local pi-package catalog for packages that satisfy the following need, and complete an integration review:`,
171
+ `**${query}**`,
172
+ "",
173
+ "Steps:",
174
+ "1. Search: run multiple jq/grep keyword passes over `~/.pi/agent/data/pi-find-packages/catalog.jsonl` against description/keywords/name, trying synonyms as needed; "
175
+ + (semanticEnabled()
176
+ ? "also run `qmd query --collection pi-pkg-readmes` for semantic search over cached READMEs (skip if empty); "
177
+ : "")
178
+ + "pick the 3-5 most relevant candidates.",
179
+ "2. Review each candidate (read-only): `npm view <pkg>` for version/deps/peer; clone or download source via the repo link; read entry points, extension points, and the README; judge maintenance activity, dependency surface, and supply-chain signals. "
180
+ + "After reviewing, save the package README to `~/.pi/agent/data/pi-find-packages/readmes/<name with / replaced by __>.md` (first line `# <name> <version> <date>`)"
181
+ + (semanticEnabled() ? ", then run `qmd index pi-pkg-readmes` to update the index" : "") + ".",
182
+ `3. Execution environment: ${iso === "docker" ? "all cloning/unpacking/source analysis must run inside the Docker container (see docker/Dockerfile.analysis); the host only receives analysis text; never run a candidate's install scripts." : "not isolated — analyze read-only on the host (see the risk note above)."}`,
183
+ "4. Criteria: feature overlap with the current setup / installed packages (Unix philosophy: features must not cross); pi compatibility (peer ranges); maintenance activity; dependency and supply-chain safety.",
184
+ "5. Output: a candidate comparison table (name/version/activity/fit/risk) + a clear recommendation with reasons. The analysis report is not install authorization — I decide whether to integrate.",
185
+ riskNote,
186
+ ].filter(Boolean).join("\n");
187
+ await pi.sendUserMessage(prompt, { deliverAs: "followUp" });
188
+ },
189
+ });
190
+ }
package/package.json ADDED
@@ -0,0 +1,41 @@
1
+ {
2
+ "name": "@nailuogg/pi-find-packages",
3
+ "version": "0.1.1",
4
+ "description": "Local catalog of the pi package ecosystem with /find-packages: offline search, cold-start data, and sandboxed source analysis for integration review",
5
+ "type": "module",
6
+ "keywords": [
7
+ "pi-package",
8
+ "pi",
9
+ "pi-extension",
10
+ "packages",
11
+ "catalog",
12
+ "discovery"
13
+ ],
14
+ "license": "MIT",
15
+ "engines": {
16
+ "node": ">=22"
17
+ },
18
+ "pi": {
19
+ "extensions": [
20
+ "./extensions/find-packages.ts"
21
+ ],
22
+ "skills": [
23
+ "./skills"
24
+ ]
25
+ },
26
+ "author": "nailuoGG",
27
+ "repository": {
28
+ "type": "git",
29
+ "url": "git+https://github.com/nailuoGG/pi-find-packages.git"
30
+ },
31
+ "bugs": {
32
+ "url": "https://github.com/nailuoGG/pi-find-packages/issues"
33
+ },
34
+ "homepage": "https://github.com/nailuoGG/pi-find-packages#readme",
35
+ "peerDependencies": {
36
+ "@earendil-works/pi-coding-agent": "*"
37
+ },
38
+ "publishConfig": {
39
+ "access": "public"
40
+ }
41
+ }
@@ -0,0 +1,76 @@
1
+ #!/usr/bin/env node
2
+ // Sync the pi-package catalog from the npm registry search API.
3
+ // Data source: https://registry.npmjs.org/-/v1/search?text=keywords:pi-package
4
+ // Output: JSONL (one package per line) at ~/.pi/agent/data/pi-find-packages/catalog.jsonl
5
+ // Usage: node sync-catalog.mjs [--out <dir>]
6
+ // --out: output directory (default: ~/.pi/agent/data/pi-find-packages; CI writes to a checkout of the data branch)
7
+
8
+ import { mkdirSync, writeFileSync, renameSync, statSync, readFileSync } from "node:fs";
9
+ import { join } from "node:path";
10
+ import { homedir } from "node:os";
11
+ import { gzipSync } from "node:zlib";
12
+ import { createHash } from "node:crypto";
13
+
14
+ const PAGE_SIZE = 250;
15
+ const REQUEST_DELAY_MS = 1000; // shared runner IPs get rate-limited fast; stay polite // be polite to the registry
16
+ const outArg = process.argv.indexOf("--out");
17
+ const OUT_DIR = outArg > -1 ? process.argv[outArg + 1] : join(homedir(), ".pi/agent/data/pi-find-packages");
18
+ const OUT_FILE = join(OUT_DIR, "catalog.jsonl");
19
+ const OUT_GZ = join(OUT_DIR, "catalog.jsonl.gz");
20
+
21
+ const seen = new Map(); // name -> record (dedupe across pages)
22
+ let total = Infinity;
23
+
24
+ async function fetchPage(from) {
25
+ const url = `https://registry.npmjs.org/-/v1/search?text=keywords:pi-package&size=${PAGE_SIZE}&from=${from}`;
26
+ for (let attempt = 0; ; attempt++) {
27
+ const res = await fetch(url, { headers: { accept: "application/json" } });
28
+ if (res.ok) return res.json();
29
+ if (attempt >= 5 || (res.status !== 429 && res.status >= 500)) {
30
+ throw new Error(`registry ${res.status} at from=${from} after ${attempt + 1} attempts`);
31
+ }
32
+ const wait = Math.min(30000, 2000 * 2 ** attempt); // 2s, 4s, 8s, 16s, 30s, 30s
33
+ process.stderr.write(`registry ${res.status} at from=${from}; retrying in ${wait / 1000}s\n`);
34
+ await new Promise((r) => setTimeout(r, wait));
35
+ }
36
+ }
37
+
38
+ function toRecord(p) {
39
+ const links = p.links || {};
40
+ return {
41
+ name: p.name,
42
+ version: p.version,
43
+ description: p.description || "",
44
+ date: p.date || "",
45
+ author: typeof p.author === "string" ? p.author : p.author?.name || "",
46
+ publisher: p.publisher?.username || "",
47
+ keywords: p.keywords || [],
48
+ npm: links.npm || `https://www.npmjs.com/package/${p.name}`,
49
+ repo: links.repository || "",
50
+ homepage: links.homepage || "",
51
+ };
52
+ }
53
+
54
+ for (let from = 0; from < total; from += PAGE_SIZE) {
55
+ const data = await fetchPage(from);
56
+ total = Math.min(data.total, 10000); // search API caps total at 10000
57
+ for (const { package: p } of data.objects) seen.set(p.name, toRecord(p));
58
+ process.stderr.write(`fetched ${seen.size}/${total}\n`);
59
+ if (from + PAGE_SIZE < total) await new Promise((r) => setTimeout(r, REQUEST_DELAY_MS));
60
+ }
61
+
62
+ const lines = [...seen.values()]
63
+ .sort((a, b) => a.name.localeCompare(b.name))
64
+ .map((r) => JSON.stringify(r));
65
+
66
+ mkdirSync(OUT_DIR, { recursive: true });
67
+ const tmp = OUT_FILE + ".tmp";
68
+ writeFileSync(tmp, lines.join("\n") + "\n");
69
+ renameSync(tmp, OUT_FILE);
70
+ writeFileSync(OUT_GZ, gzipSync(Buffer.from(lines.join("\n") + "\n")));
71
+
72
+ writeFileSync(OUT_GZ + ".sha256", createHash("sha256").update(readFileSync(OUT_GZ)).digest("hex") + "\n");
73
+
74
+ const bytes = statSync(OUT_FILE).size;
75
+ console.log(`done: ${lines.length} packages, ${(bytes / 1e6).toFixed(1)} MB -> ${OUT_FILE}`);
76
+ console.log(`gz: ${(statSync(OUT_GZ).size / 1e6).toFixed(1)} MB -> ${OUT_GZ}`);
@@ -0,0 +1,37 @@
1
+ # find-packages: pi package catalog search & integration review
2
+
3
+ Maintains a local catalog of the pi package ecosystem (same source as pi.dev/packages: npm `keywords:pi-package`) for offline/semantic package discovery, plus a standard integration-evaluation workflow.
4
+
5
+ ## Data
6
+
7
+ - Catalog file: `~/.pi/agent/data/pi-find-packages/catalog.jsonl` (one package per line: name/version/description/date/author/keywords/npm/repo)
8
+ - Refresh (pick the first that works):
9
+ 1. `/find-packages update` — pulls the latest snapshot from the jsDelivr `data` branch (falls back to the npmmirror tarball of the latest published version); checksum-verified, atomic replace
10
+ 2. `node <package-dir>/scripts/sync-catalog.mjs` — full rebuild from the npm registry search API (~40 requests). Only run when the file is older than 30 days or the user asks; avoid hammering the registry.
11
+
12
+ ## Search methods
13
+
14
+ 1. Lexical: `jq -r 'select(.description|test("keyword";"i")) | [.name,.version,.description] | @tsv' catalog.jsonl` — try multiple synonym groups (English-first; cover channel words like remote/telegram/discord/web).
15
+ 2. Semantic (optional): `qmd query --collection pi-pkg-readmes "need description"` — searches cached READMEs of previously reviewed packages (grows over time; skip when the cache is empty). Only use when the `qmd` binary is available and not disabled via `"semantic":"off"` in `config.json`; otherwise skip this step and rely on lexical search.
16
+ 3. When both come up empty, fall back to `npm search` and note the catalog may be stale.
17
+
18
+ ## README cache (written as a side effect of review)
19
+
20
+ After reviewing a candidate, save its README to `~/.pi/agent/data/pi-find-packages/readmes/<name with / replaced by __>.md`, first line `# <name> <version> <date>`; then run `qmd index pi-pkg-readmes` to update the index. **Never bulk-fetch all READMEs** — the corpus grows with real reviews.
21
+
22
+ ## Integration review (check every candidate)
23
+
24
+ - **Feature overlap**: does it overlap installed packages (`packages` arrays in `~/.pi/agent/settings.json` and the project `.pi/settings.json`) or pi built-ins? No overlap is a hard rule.
25
+ - **Compatibility**: does `npm view <pkg> peerDependencies` cover the current pi version?
26
+ - **Activity**: latest publish date, latest repo commit.
27
+ - **Supply chain**: author/maintainers, dependency count, install/preinstall scripts in package.json, any data-exfiltration paths.
28
+
29
+ ## Execution environment
30
+
31
+ - **Source analysis must run inside Docker by default** (clone/unpack candidate code via `docker/Dockerfile.analysis`: node24-slim + git + ripgrep, no credentials). The host only receives analysis text. **Never execute a candidate's install scripts or build artifacts.**
32
+ - `"isolation":"off"` in `data/pi-find-packages/config.json` disables isolation — a risk warning is shown on every use; not recommended.
33
+ - Shallow lookups (`npm view`, registry JSON) do not need the container.
34
+
35
+ ## Output convention
36
+
37
+ Candidate comparison table (name/version/last publish/activity/fit/risk) + a clear recommendation with reasons. **An analysis report is not install authorization** — the user decides whether to `pi install`.