@nailuogg/pi-find-packages 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/publish.yml +40 -0
- package/.github/workflows/update-data.yml +45 -0
- package/LICENSE +21 -0
- package/README.md +68 -0
- package/data/catalog.jsonl.gz +0 -0
- package/docker/Dockerfile.analysis +13 -0
- package/extensions/find-packages.ts +190 -0
- package/package.json +41 -0
- package/scripts/sync-catalog.mjs +76 -0
- package/skills/find-packages/SKILL.md +37 -0
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
name: publish
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags: ["v*"]
|
|
6
|
+
|
|
7
|
+
permissions:
|
|
8
|
+
contents: write # create the GitHub release
|
|
9
|
+
id-token: write # reserved for a future Trusted-Publishing (OIDC) switch
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
publish:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v4
|
|
16
|
+
|
|
17
|
+
- uses: actions/setup-node@v4
|
|
18
|
+
with:
|
|
19
|
+
node-version: 24
|
|
20
|
+
registry-url: https://registry.npmjs.org/
|
|
21
|
+
# reads NODE_AUTH_TOKEN from env to write ~/.npmrc
|
|
22
|
+
|
|
23
|
+
- name: Verify version matches tag
|
|
24
|
+
run: |
|
|
25
|
+
PKG_VERSION="${GITHUB_REF_NAME#v}"
|
|
26
|
+
TAG_VERSION="$(node -p "require('./package.json').version")"
|
|
27
|
+
if [ "${PKG_VERSION}" != "${TAG_VERSION}" ]; then
|
|
28
|
+
echo "::error::tag ${GITHUB_REF_NAME} != package.json version ${TAG_VERSION}"
|
|
29
|
+
exit 1
|
|
30
|
+
fi
|
|
31
|
+
|
|
32
|
+
- name: Publish
|
|
33
|
+
run: npm publish
|
|
34
|
+
env:
|
|
35
|
+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
36
|
+
|
|
37
|
+
- name: Create GitHub release
|
|
38
|
+
run: gh release create "${GITHUB_REF_NAME}" --verify-tag --generate-notes
|
|
39
|
+
env:
|
|
40
|
+
GH_TOKEN: ${{ github.token }}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
name: update-data
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
schedule:
|
|
5
|
+
- cron: "17 3 * * *" # daily, off the hour
|
|
6
|
+
workflow_dispatch:
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: write
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
refresh:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v4
|
|
16
|
+
|
|
17
|
+
- uses: actions/setup-node@v4
|
|
18
|
+
with:
|
|
19
|
+
node-version: 24
|
|
20
|
+
|
|
21
|
+
- name: Fetch latest catalog
|
|
22
|
+
run: node scripts/sync-catalog.mjs --out "${RUNNER_TEMP}/catalog"
|
|
23
|
+
|
|
24
|
+
- name: Skip if unchanged
|
|
25
|
+
id: diff
|
|
26
|
+
run: |
|
|
27
|
+
NEW="${RUNNER_TEMP}/catalog/catalog.jsonl.gz"
|
|
28
|
+
OLD="$(curl -fsSL --max-time 30 https://cdn.jsdelivr.net/gh/${GITHUB_REPOSITORY}@data/data/catalog.jsonl.gz 2>/dev/null | sha256sum | cut -d' ' -f1 || true)"
|
|
29
|
+
CUR="$(sha256sum "${NEW}" | cut -d' ' -f1)"
|
|
30
|
+
echo "changed=$([[ "${CUR}" != "${OLD}" ]] && echo true || echo false)" >> "$GITHUB_OUTPUT"
|
|
31
|
+
|
|
32
|
+
- name: Push to data branch
|
|
33
|
+
if: steps.diff.outputs.changed == 'true'
|
|
34
|
+
run: |
|
|
35
|
+
git config user.name "github-actions[bot]"
|
|
36
|
+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
|
37
|
+
git fetch origin data || true
|
|
38
|
+
git checkout -B data origin/data 2>/dev/null || git checkout -B data
|
|
39
|
+
mkdir -p data
|
|
40
|
+
cp "${RUNNER_TEMP}/catalog/catalog.jsonl" data/
|
|
41
|
+
cp "${RUNNER_TEMP}/catalog/catalog.jsonl.gz" data/
|
|
42
|
+
cp "${RUNNER_TEMP}/catalog/catalog.jsonl.gz.sha256" data/
|
|
43
|
+
git add data/
|
|
44
|
+
git commit -qm "chore(data): refresh catalog ($(date -u +%Y-%m-%d))"
|
|
45
|
+
git push origin data
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 nailuoGG
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# pi-find-packages
|
|
2
|
+
|
|
3
|
+
A [pi](https://pi.dev) extension providing a local catalog of the pi package ecosystem plus a `/find-packages` integration-review workflow.
|
|
4
|
+
|
|
5
|
+
## Features
|
|
6
|
+
|
|
7
|
+
- **Offline catalog**: syncs the full `keywords:pi-package` corpus (~5k+ packages) from npm to local JSONL; search with jq/grep over descriptions, names, and keywords — no dependence on npm's keyword-matched search
|
|
8
|
+
- **`/find-packages <need>`**: retrieve candidates → read-only source analysis in a sandbox → comparison table and recommendation against criteria (feature overlap / peer compatibility / maintenance activity / supply-chain signals)
|
|
9
|
+
- **Cold start**: the package ships a bundled catalog snapshot (`data/catalog.jsonl.gz`); install and use immediately with zero network requests on first run
|
|
10
|
+
- **Docker isolation by default**: candidate repos are cloned and unpacked inside a credential-free, non-root container; isolation can be disabled explicitly (a risk warning is shown on every use — not recommended)
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
pi install npm:@nailuogg/pi-find-packages
|
|
16
|
+
pi install git:github.com/nailuoGG/pi-find-packages
|
|
17
|
+
pi install /path/to/pi-find-packages # local trial
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Data directory
|
|
21
|
+
|
|
22
|
+
`<PI_CODING_AGENT_DIR>/data/pi-find-packages/` (default `~/.pi/agent/data/pi-find-packages/`)
|
|
23
|
+
|
|
24
|
+
- `catalog.jsonl` — catalog data (one package per line: name/version/description/date/author/keywords/repo)
|
|
25
|
+
- `config.json` — `{"isolation": "docker" | "off", "semantic": "auto" | "on" | "off"}`
|
|
26
|
+
- `isolation` defaults to `docker` (sandboxed source analysis)
|
|
27
|
+
- `semantic` controls semantic search over lazily-cached READMEs (via [qmd](https://github.com/tobi/qmd)); defaults to `auto`: enabled automatically when the `qmd` binary is present, force with `"on"`, disable with `"off"`
|
|
28
|
+
|
|
29
|
+
## Refreshing the catalog
|
|
30
|
+
|
|
31
|
+
Preferred, in order:
|
|
32
|
+
|
|
33
|
+
1. `/find-packages update` — pulls the latest snapshot from the jsDelivr `data` branch
|
|
34
|
+
(`cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data`), falls back to the npmmirror
|
|
35
|
+
tarball of the latest published version. Checksum-verified, atomic replace.
|
|
36
|
+
2. Full rebuild from the npm registry search API (~40 requests):
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
node <package-dir>/scripts/sync-catalog.mjs # default data dir
|
|
40
|
+
node scripts/sync-catalog.mjs --out /tmp/catalog # custom output dir (used by CI)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Once every ≥30 days is plenty — please avoid hammering the npm registry.
|
|
44
|
+
|
|
45
|
+
A GitHub Actions workflow (`update-data.yml`) refreshes the `data` branch daily when the
|
|
46
|
+
corpus changes, which is what the jsDelivr channel serves. The npmmirror channel tracks
|
|
47
|
+
the latest npm publish and therefore updates on release cadence.
|
|
48
|
+
|
|
49
|
+
## Analysis sandbox
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
docker build -t pi-find-packages-analysis -f docker/Dockerfile.analysis docker
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
The image contains no pi and no credentials: it exists solely to clone/unpack/read third-party source. Never execute a candidate package's install scripts or build artifacts in any environment.
|
|
56
|
+
|
|
57
|
+
## Releasing
|
|
58
|
+
|
|
59
|
+
Publishing is automated: push a tag `v<version>` matching `package.json`, and the
|
|
60
|
+
`publish.yml` workflow verifies the version, publishes `@nailuogg/pi-find-packages`
|
|
61
|
+
to npm, and creates a GitHub release. The `NPM_TOKEN` repository secret must hold a
|
|
62
|
+
granular npm access token scoped to the `@nailuogg` packages (npmjs.com → Access Keys).
|
|
63
|
+
|
|
64
|
+
## Security boundaries
|
|
65
|
+
|
|
66
|
+
- An analysis report is **not** install authorization — whether to `pi install` is always the user's decision
|
|
67
|
+
- `isolation: off` injects a risk warning on every use; third-party package source may contain malicious logic
|
|
68
|
+
- Planned: update channel via GitHub tag + jsDelivr CDN with checksum verification
|
|
Binary file
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
FROM node:24-bookworm-slim
|
|
2
|
+
|
|
3
|
+
RUN apt-get update \
|
|
4
|
+
&& apt-get install -y --no-install-recommends bash ca-certificates git ripgrep curl jq \
|
|
5
|
+
&& rm -rf /var/lib/apt/lists/*
|
|
6
|
+
|
|
7
|
+
# No pi, no credentials: this image only clones/unpacks and reads third-party source.
|
|
8
|
+
RUN useradd -m -s /bin/bash analyst
|
|
9
|
+
RUN mkdir -p /analysis && chown analyst:analyst /analysis
|
|
10
|
+
WORKDIR /analysis
|
|
11
|
+
USER analyst
|
|
12
|
+
|
|
13
|
+
ENTRYPOINT ["bash"]
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* /find-packages — search the local pi-package catalog and kick off guided analysis.
|
|
3
|
+
*
|
|
4
|
+
* Data dir: <agentDir>/data/pi-find-packages/catalog.jsonl
|
|
5
|
+
* Cold start: extracts the bundled data/catalog.jsonl.gz on first run.
|
|
6
|
+
* Config (config.json in the data dir):
|
|
7
|
+
* isolation: "docker" (default) | "off" — source-analysis execution environment
|
|
8
|
+
* semantic: "auto" (default) | "on" | "off" — qmd-backed semantic search over cached READMEs
|
|
9
|
+
* Subcommand: /find-packages update — refresh the catalog from jsDelivr (data branch),
|
|
10
|
+
* then the npmmirror tarball of the latest published version; local sync script as last resort.
|
|
11
|
+
*/
|
|
12
|
+
import { existsSync, mkdirSync, readFileSync, writeFileSync, renameSync } from "node:fs";
|
|
13
|
+
import { spawnSync } from "node:child_process";
|
|
14
|
+
import { gunzipSync } from "node:zlib";
|
|
15
|
+
import { createHash } from "node:crypto";
|
|
16
|
+
import { tmpdir } from "node:os";
|
|
17
|
+
import { join, dirname } from "node:path";
|
|
18
|
+
import { fileURLToPath } from "node:url";
|
|
19
|
+
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
|
|
20
|
+
|
|
21
|
+
const JSDELIVR_GZ = "https://cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data/data/catalog.jsonl.gz";
|
|
22
|
+
const JSDELIVR_SHA = "https://cdn.jsdelivr.net/gh/nailuoGG/pi-find-packages@data/data/catalog.jsonl.gz.sha256";
|
|
23
|
+
const NPMIRROR_TARBALL = "https://registry.npmmirror.com/@nailuogg%2Fpi-find-packages/latest";
|
|
24
|
+
|
|
25
|
+
export default function activate(pi: ExtensionAPI) {
|
|
26
|
+
// PI_CODING_AGENT_DIR is the official config-dir override (docs/environment-variables.md).
|
|
27
|
+
const agentDir = process.env.PI_CODING_AGENT_DIR || join(process.env.HOME ?? "", ".pi/agent");
|
|
28
|
+
const dataDir = join(agentDir, "data/pi-find-packages");
|
|
29
|
+
const catalog = join(dataDir, "catalog.jsonl");
|
|
30
|
+
const configFile = join(dataDir, "config.json");
|
|
31
|
+
|
|
32
|
+
// Cold start: extract bundled snapshot if catalog missing.
|
|
33
|
+
const pkgDir = dirname(dirname(fileURLToPath(import.meta.url))); // extensions/../ = package root
|
|
34
|
+
const bundled = join(pkgDir, "data/catalog.jsonl.gz");
|
|
35
|
+
if (!existsSync(catalog)) {
|
|
36
|
+
mkdirSync(dataDir, { recursive: true });
|
|
37
|
+
if (existsSync(bundled)) {
|
|
38
|
+
writeFileSync(catalog, gunzipSync(readFileSync(bundled)));
|
|
39
|
+
pi.appendEntry("find-packages", { event: "cold-start-extract", from: bundled });
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
function readConfig(): Record<string, unknown> {
|
|
44
|
+
try {
|
|
45
|
+
return JSON.parse(readFileSync(configFile, "utf8"));
|
|
46
|
+
} catch {
|
|
47
|
+
return {};
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
function isolation(): "docker" | "off" {
|
|
52
|
+
return readConfig().isolation === "off" ? "off" : "docker"; // secure default
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
// Semantic search via qmd: "auto" (default) = enabled iff qmd binary exists; "on" forces; "off" disables.
|
|
56
|
+
let qmdAvailable: boolean | undefined;
|
|
57
|
+
function semanticEnabled() {
|
|
58
|
+
const pref = readConfig().semantic;
|
|
59
|
+
if (pref === "off") return false;
|
|
60
|
+
if (qmdAvailable === undefined) {
|
|
61
|
+
try {
|
|
62
|
+
qmdAvailable = spawnSync("qmd", ["--version"], { timeout: 5000 }).status === 0;
|
|
63
|
+
} catch {
|
|
64
|
+
qmdAvailable = false;
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
return qmdAvailable;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
async function fetchText(url: string, timeoutMs = 15000): Promise<string | undefined> {
|
|
71
|
+
try {
|
|
72
|
+
const res = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) });
|
|
73
|
+
if (!res.ok) return undefined;
|
|
74
|
+
return await res.text();
|
|
75
|
+
} catch {
|
|
76
|
+
return undefined;
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
async function fetchBuffer(url: string, timeoutMs = 30000): Promise<Buffer | undefined> {
|
|
81
|
+
try {
|
|
82
|
+
const res = await fetch(url, { signal: AbortSignal.timeout(timeoutMs) });
|
|
83
|
+
if (!res.ok) return undefined;
|
|
84
|
+
return Buffer.from(await res.arrayBuffer());
|
|
85
|
+
} catch {
|
|
86
|
+
return undefined;
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
function sha256(buf: Buffer): string {
|
|
91
|
+
return createHash("sha256").update(buf).digest("hex");
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
function installCatalog(buf: Buffer, ctx): boolean {
|
|
95
|
+
const tmp = join(tmpdir(), `catalog.jsonl.${Date.now()}`);
|
|
96
|
+
try {
|
|
97
|
+
const jsonl = gunzipSync(buf);
|
|
98
|
+
JSON.parse(jsonl.toString("utf8").split("\n")[0]); // sanity: first line parses
|
|
99
|
+
writeFileSync(tmp, jsonl);
|
|
100
|
+
renameSync(tmp, catalog);
|
|
101
|
+
return true;
|
|
102
|
+
} catch {
|
|
103
|
+
try { spawnSync("rm", ["-f", tmp]); } catch {}
|
|
104
|
+
ctx.ui.notify("Downloaded catalog failed validation; keeping the existing file.", "error");
|
|
105
|
+
return false;
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
async function updateCatalog(ctx): Promise<void> {
|
|
110
|
+
// 1) jsDelivr data branch (freshest; updated by CI)
|
|
111
|
+
const gz = await fetchBuffer(JSDELIVR_GZ);
|
|
112
|
+
const expected = (await fetchText(JSDELIVR_SHA))?.trim();
|
|
113
|
+
if (gz && expected && sha256(gz) === expected.split(/\s+/)[0]) {
|
|
114
|
+
if (installCatalog(gz, ctx)) ctx.ui.notify("Catalog updated from jsDelivr (data branch).", "info");
|
|
115
|
+
return;
|
|
116
|
+
}
|
|
117
|
+
// 2) npmmirror tarball of the latest published version (freshness = publish cadence)
|
|
118
|
+
try {
|
|
119
|
+
const meta = await (await fetch(NPMIRROR_TARBALL, { signal: AbortSignal.timeout(15000) })).json();
|
|
120
|
+
const tarballUrl: string | undefined = meta?.dist?.tarball;
|
|
121
|
+
if (tarballUrl) {
|
|
122
|
+
const res = await fetch(tarballUrl, { signal: AbortSignal.timeout(60000) });
|
|
123
|
+
if (res.ok) {
|
|
124
|
+
const buf = Buffer.from(await res.arrayBuffer());
|
|
125
|
+
const proc = spawnSync("tar", ["-xzOf", "-", "package/data/catalog.jsonl.gz"], { input: buf });
|
|
126
|
+
if (proc.status === 0 && proc.stdout?.length) {
|
|
127
|
+
if (installCatalog(Buffer.from(proc.stdout), ctx)) {
|
|
128
|
+
ctx.ui.notify("Catalog updated from npmmirror (latest published version).", "info");
|
|
129
|
+
return;
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
} catch {}
|
|
135
|
+
// 3) fall back to the local sync script
|
|
136
|
+
ctx.ui.notify(
|
|
137
|
+
"Both CDNs unavailable or checksum mismatch. Rebuild locally with:\n"
|
|
138
|
+
+ ` node ${join(pkgDir, "scripts/sync-catalog.mjs")}`,
|
|
139
|
+
"warning",
|
|
140
|
+
);
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
pi.registerCommand("find-packages", {
|
|
144
|
+
description: "Search local pi-package catalog for packages matching a need; `update` refreshes the catalog",
|
|
145
|
+
handler: async (args, ctx) => {
|
|
146
|
+
const query = (args ?? "").trim();
|
|
147
|
+
if (query === "update") {
|
|
148
|
+
await updateCatalog(ctx);
|
|
149
|
+
return;
|
|
150
|
+
}
|
|
151
|
+
if (!query) {
|
|
152
|
+
ctx.ui.notify("Usage: /find-packages <need> — e.g. /find-packages control sessions remotely; or /find-packages update", "warning");
|
|
153
|
+
return;
|
|
154
|
+
}
|
|
155
|
+
if (!existsSync(catalog)) {
|
|
156
|
+
ctx.ui.notify(`catalog.jsonl not found. Run: /find-packages update — or: node ${join(pkgDir, "scripts/sync-catalog.mjs")}`, "error");
|
|
157
|
+
return;
|
|
158
|
+
}
|
|
159
|
+
const iso = isolation();
|
|
160
|
+
const riskNote =
|
|
161
|
+
iso === "off"
|
|
162
|
+
? [
|
|
163
|
+
"",
|
|
164
|
+
"⚠️ **Isolation disabled (isolation: off)**: candidate source will be cloned/unpacked on the host.",
|
|
165
|
+
"Third-party packages may contain malicious install scripts or build logic; host-side analysis is not sandboxed.",
|
|
166
|
+
"This mode is not recommended. Delete data/pi-find-packages/config.json or set isolation:\"docker\" to restore the default.",
|
|
167
|
+
].join("\n")
|
|
168
|
+
: "";
|
|
169
|
+
const prompt = [
|
|
170
|
+
`Search the local pi-package catalog for packages that satisfy the following need, and complete an integration review:`,
|
|
171
|
+
`**${query}**`,
|
|
172
|
+
"",
|
|
173
|
+
"Steps:",
|
|
174
|
+
"1. Search: run multiple jq/grep keyword passes over `~/.pi/agent/data/pi-find-packages/catalog.jsonl` against description/keywords/name, trying synonyms as needed; "
|
|
175
|
+
+ (semanticEnabled()
|
|
176
|
+
? "also run `qmd query --collection pi-pkg-readmes` for semantic search over cached READMEs (skip if empty); "
|
|
177
|
+
: "")
|
|
178
|
+
+ "pick the 3-5 most relevant candidates.",
|
|
179
|
+
"2. Review each candidate (read-only): `npm view <pkg>` for version/deps/peer; clone or download source via the repo link; read entry points, extension points, and the README; judge maintenance activity, dependency surface, and supply-chain signals. "
|
|
180
|
+
+ "After reviewing, save the package README to `~/.pi/agent/data/pi-find-packages/readmes/<name with / replaced by __>.md` (first line `# <name> <version> <date>`)"
|
|
181
|
+
+ (semanticEnabled() ? ", then run `qmd index pi-pkg-readmes` to update the index" : "") + ".",
|
|
182
|
+
`3. Execution environment: ${iso === "docker" ? "all cloning/unpacking/source analysis must run inside the Docker container (see docker/Dockerfile.analysis); the host only receives analysis text; never run a candidate's install scripts." : "not isolated — analyze read-only on the host (see the risk note above)."}`,
|
|
183
|
+
"4. Criteria: feature overlap with the current setup / installed packages (Unix philosophy: features must not cross); pi compatibility (peer ranges); maintenance activity; dependency and supply-chain safety.",
|
|
184
|
+
"5. Output: a candidate comparison table (name/version/activity/fit/risk) + a clear recommendation with reasons. The analysis report is not install authorization — I decide whether to integrate.",
|
|
185
|
+
riskNote,
|
|
186
|
+
].filter(Boolean).join("\n");
|
|
187
|
+
await pi.sendUserMessage(prompt, { deliverAs: "followUp" });
|
|
188
|
+
},
|
|
189
|
+
});
|
|
190
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@nailuogg/pi-find-packages",
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"description": "Local catalog of the pi package ecosystem with /find-packages: offline search, cold-start data, and sandboxed source analysis for integration review",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"keywords": [
|
|
7
|
+
"pi-package",
|
|
8
|
+
"pi",
|
|
9
|
+
"pi-extension",
|
|
10
|
+
"packages",
|
|
11
|
+
"catalog",
|
|
12
|
+
"discovery"
|
|
13
|
+
],
|
|
14
|
+
"license": "MIT",
|
|
15
|
+
"engines": {
|
|
16
|
+
"node": ">=22"
|
|
17
|
+
},
|
|
18
|
+
"pi": {
|
|
19
|
+
"extensions": [
|
|
20
|
+
"./extensions/find-packages.ts"
|
|
21
|
+
],
|
|
22
|
+
"skills": [
|
|
23
|
+
"./skills"
|
|
24
|
+
]
|
|
25
|
+
},
|
|
26
|
+
"author": "nailuoGG",
|
|
27
|
+
"repository": {
|
|
28
|
+
"type": "git",
|
|
29
|
+
"url": "git+https://github.com/nailuoGG/pi-find-packages.git"
|
|
30
|
+
},
|
|
31
|
+
"bugs": {
|
|
32
|
+
"url": "https://github.com/nailuoGG/pi-find-packages/issues"
|
|
33
|
+
},
|
|
34
|
+
"homepage": "https://github.com/nailuoGG/pi-find-packages#readme",
|
|
35
|
+
"peerDependencies": {
|
|
36
|
+
"@earendil-works/pi-coding-agent": "*"
|
|
37
|
+
},
|
|
38
|
+
"publishConfig": {
|
|
39
|
+
"access": "public"
|
|
40
|
+
}
|
|
41
|
+
}
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Sync the pi-package catalog from the npm registry search API.
|
|
3
|
+
// Data source: https://registry.npmjs.org/-/v1/search?text=keywords:pi-package
|
|
4
|
+
// Output: JSONL (one package per line) at ~/.pi/agent/data/pi-find-packages/catalog.jsonl
|
|
5
|
+
// Usage: node sync-catalog.mjs [--out <dir>]
|
|
6
|
+
// --out: output directory (default: ~/.pi/agent/data/pi-find-packages; CI writes to a checkout of the data branch)
|
|
7
|
+
|
|
8
|
+
import { mkdirSync, writeFileSync, renameSync, statSync, readFileSync } from "node:fs";
|
|
9
|
+
import { join } from "node:path";
|
|
10
|
+
import { homedir } from "node:os";
|
|
11
|
+
import { gzipSync } from "node:zlib";
|
|
12
|
+
import { createHash } from "node:crypto";
|
|
13
|
+
|
|
14
|
+
const PAGE_SIZE = 250;
|
|
15
|
+
const REQUEST_DELAY_MS = 1000; // shared runner IPs get rate-limited fast; stay polite // be polite to the registry
|
|
16
|
+
const outArg = process.argv.indexOf("--out");
|
|
17
|
+
const OUT_DIR = outArg > -1 ? process.argv[outArg + 1] : join(homedir(), ".pi/agent/data/pi-find-packages");
|
|
18
|
+
const OUT_FILE = join(OUT_DIR, "catalog.jsonl");
|
|
19
|
+
const OUT_GZ = join(OUT_DIR, "catalog.jsonl.gz");
|
|
20
|
+
|
|
21
|
+
const seen = new Map(); // name -> record (dedupe across pages)
|
|
22
|
+
let total = Infinity;
|
|
23
|
+
|
|
24
|
+
async function fetchPage(from) {
|
|
25
|
+
const url = `https://registry.npmjs.org/-/v1/search?text=keywords:pi-package&size=${PAGE_SIZE}&from=${from}`;
|
|
26
|
+
for (let attempt = 0; ; attempt++) {
|
|
27
|
+
const res = await fetch(url, { headers: { accept: "application/json" } });
|
|
28
|
+
if (res.ok) return res.json();
|
|
29
|
+
if (attempt >= 5 || (res.status !== 429 && res.status >= 500)) {
|
|
30
|
+
throw new Error(`registry ${res.status} at from=${from} after ${attempt + 1} attempts`);
|
|
31
|
+
}
|
|
32
|
+
const wait = Math.min(30000, 2000 * 2 ** attempt); // 2s, 4s, 8s, 16s, 30s, 30s
|
|
33
|
+
process.stderr.write(`registry ${res.status} at from=${from}; retrying in ${wait / 1000}s\n`);
|
|
34
|
+
await new Promise((r) => setTimeout(r, wait));
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
function toRecord(p) {
|
|
39
|
+
const links = p.links || {};
|
|
40
|
+
return {
|
|
41
|
+
name: p.name,
|
|
42
|
+
version: p.version,
|
|
43
|
+
description: p.description || "",
|
|
44
|
+
date: p.date || "",
|
|
45
|
+
author: typeof p.author === "string" ? p.author : p.author?.name || "",
|
|
46
|
+
publisher: p.publisher?.username || "",
|
|
47
|
+
keywords: p.keywords || [],
|
|
48
|
+
npm: links.npm || `https://www.npmjs.com/package/${p.name}`,
|
|
49
|
+
repo: links.repository || "",
|
|
50
|
+
homepage: links.homepage || "",
|
|
51
|
+
};
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
for (let from = 0; from < total; from += PAGE_SIZE) {
|
|
55
|
+
const data = await fetchPage(from);
|
|
56
|
+
total = Math.min(data.total, 10000); // search API caps total at 10000
|
|
57
|
+
for (const { package: p } of data.objects) seen.set(p.name, toRecord(p));
|
|
58
|
+
process.stderr.write(`fetched ${seen.size}/${total}\n`);
|
|
59
|
+
if (from + PAGE_SIZE < total) await new Promise((r) => setTimeout(r, REQUEST_DELAY_MS));
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
const lines = [...seen.values()]
|
|
63
|
+
.sort((a, b) => a.name.localeCompare(b.name))
|
|
64
|
+
.map((r) => JSON.stringify(r));
|
|
65
|
+
|
|
66
|
+
mkdirSync(OUT_DIR, { recursive: true });
|
|
67
|
+
const tmp = OUT_FILE + ".tmp";
|
|
68
|
+
writeFileSync(tmp, lines.join("\n") + "\n");
|
|
69
|
+
renameSync(tmp, OUT_FILE);
|
|
70
|
+
writeFileSync(OUT_GZ, gzipSync(Buffer.from(lines.join("\n") + "\n")));
|
|
71
|
+
|
|
72
|
+
writeFileSync(OUT_GZ + ".sha256", createHash("sha256").update(readFileSync(OUT_GZ)).digest("hex") + "\n");
|
|
73
|
+
|
|
74
|
+
const bytes = statSync(OUT_FILE).size;
|
|
75
|
+
console.log(`done: ${lines.length} packages, ${(bytes / 1e6).toFixed(1)} MB -> ${OUT_FILE}`);
|
|
76
|
+
console.log(`gz: ${(statSync(OUT_GZ).size / 1e6).toFixed(1)} MB -> ${OUT_GZ}`);
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# find-packages: pi package catalog search & integration review
|
|
2
|
+
|
|
3
|
+
Maintains a local catalog of the pi package ecosystem (same source as pi.dev/packages: npm `keywords:pi-package`) for offline/semantic package discovery, plus a standard integration-evaluation workflow.
|
|
4
|
+
|
|
5
|
+
## Data
|
|
6
|
+
|
|
7
|
+
- Catalog file: `~/.pi/agent/data/pi-find-packages/catalog.jsonl` (one package per line: name/version/description/date/author/keywords/npm/repo)
|
|
8
|
+
- Refresh (pick the first that works):
|
|
9
|
+
1. `/find-packages update` — pulls the latest snapshot from the jsDelivr `data` branch (falls back to the npmmirror tarball of the latest published version); checksum-verified, atomic replace
|
|
10
|
+
2. `node <package-dir>/scripts/sync-catalog.mjs` — full rebuild from the npm registry search API (~40 requests). Only run when the file is older than 30 days or the user asks; avoid hammering the registry.
|
|
11
|
+
|
|
12
|
+
## Search methods
|
|
13
|
+
|
|
14
|
+
1. Lexical: `jq -r 'select(.description|test("keyword";"i")) | [.name,.version,.description] | @tsv' catalog.jsonl` — try multiple synonym groups (English-first; cover channel words like remote/telegram/discord/web).
|
|
15
|
+
2. Semantic (optional): `qmd query --collection pi-pkg-readmes "need description"` — searches cached READMEs of previously reviewed packages (grows over time; skip when the cache is empty). Only use when the `qmd` binary is available and not disabled via `"semantic":"off"` in `config.json`; otherwise skip this step and rely on lexical search.
|
|
16
|
+
3. When both come up empty, fall back to `npm search` and note the catalog may be stale.
|
|
17
|
+
|
|
18
|
+
## README cache (written as a side effect of review)
|
|
19
|
+
|
|
20
|
+
After reviewing a candidate, save its README to `~/.pi/agent/data/pi-find-packages/readmes/<name with / replaced by __>.md`, first line `# <name> <version> <date>`; then run `qmd index pi-pkg-readmes` to update the index. **Never bulk-fetch all READMEs** — the corpus grows with real reviews.
|
|
21
|
+
|
|
22
|
+
## Integration review (check every candidate)
|
|
23
|
+
|
|
24
|
+
- **Feature overlap**: does it overlap installed packages (`packages` arrays in `~/.pi/agent/settings.json` and the project `.pi/settings.json`) or pi built-ins? No overlap is a hard rule.
|
|
25
|
+
- **Compatibility**: does `npm view <pkg> peerDependencies` cover the current pi version?
|
|
26
|
+
- **Activity**: latest publish date, latest repo commit.
|
|
27
|
+
- **Supply chain**: author/maintainers, dependency count, install/preinstall scripts in package.json, any data-exfiltration paths.
|
|
28
|
+
|
|
29
|
+
## Execution environment
|
|
30
|
+
|
|
31
|
+
- **Source analysis must run inside Docker by default** (clone/unpack candidate code via `docker/Dockerfile.analysis`: node24-slim + git + ripgrep, no credentials). The host only receives analysis text. **Never execute a candidate's install scripts or build artifacts.**
|
|
32
|
+
- `"isolation":"off"` in `data/pi-find-packages/config.json` disables isolation — a risk warning is shown on every use; not recommended.
|
|
33
|
+
- Shallow lookups (`npm view`, registry JSON) do not need the container.
|
|
34
|
+
|
|
35
|
+
## Output convention
|
|
36
|
+
|
|
37
|
+
Candidate comparison table (name/version/last publish/activity/fit/risk) + a clear recommendation with reasons. **An analysis report is not install authorization** — the user decides whether to `pi install`.
|