skillcellar 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -3
- package/package.json +1 -1
- package/src/cli.mjs +92 -7
- package/src/manifests.mjs +290 -0
- package/src/tech.mjs +242 -0
package/README.md
CHANGED
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
Keeps your [SkillCellar](https://skillcellar.com) library's list of what a computer has for Claude Code, globally and in every project: skills, plugins, MCP servers, memory files and agents, by name and hash. It also hands Claude Code the tasks you queue for that computer on the site, applies playbooks, backs the computer up to a private GitHub repository you own, restores another computer's backup, and tells you which of its skills and plugins are outdated. Node 20 or later, no dependencies; `backup` and `restore` need git. Run it with `npx skillcellar <command>`, or install it with `npm install -g skillcellar`.
|
|
4
4
|
|
|
5
5
|
1. Create an API key at https://skillcellar.com/dashboard/keys, one per computer.
|
|
6
|
-
2. `npx skillcellar login`: paste the key, then choose which of your devices this computer is, or name a new one. `--device <name>` chooses without asking. It then offers to install the skillcellar skill for Claude Code, in `~/.claude/skills/skillcellar
|
|
7
|
-
3. `npx skillcellar sync`: reports the whole computer in one request: what it has globally, then every project Claude Code knows, and every clone a restore wrote into. Each sync replaces what the last one reported from this computer, so a project you deleted, or one Claude Code no longer lists, leaves this computer's list on the site, and the site once no computer has anything in it; marks you once made by hand on the site stay. `--project <name>`, run inside a project's folder, reports only that project (every checkout of it Claude Code knows) and global, under that name, and leaves the others as they are. `--dry-run` prints what would be sent and sends nothing. The site then closes the tasks it finds done, and the sync lists them.
|
|
6
|
+
2. `npx skillcellar login`: paste the key, then choose which of your devices this computer is, or name a new one. `--device <name>` chooses without asking. It then offers to install the skillcellar skill for Claude Code, in `~/.claude/skills/skillcellar`, and asks, once, whether to report each project's technologies (see [Technologies](#technologies)).
|
|
7
|
+
3. `npx skillcellar sync`: reports the whole computer in one request: what it has globally, then every project Claude Code knows, and every clone a restore wrote into. Each sync replaces what the last one reported from this computer, so a project you deleted, or one Claude Code no longer lists, leaves this computer's list on the site, and the site once no computer has anything in it; marks you once made by hand on the site stay. `--project <name>`, run inside a project's folder, reports only that project (every checkout of it Claude Code knows) and global, under that name, and leaves the others as they are. `--dry-run` prints what would be sent and sends nothing. The site then closes the tasks it finds done, and the sync lists them. A sync at a terminal asks about technologies if `login` did not.
|
|
8
8
|
|
|
9
9
|
## Tasks
|
|
10
10
|
|
|
@@ -27,6 +27,18 @@ Both read this computer as `sync` does and send that list as `sync` sends it (na
|
|
|
27
27
|
|
|
28
28
|
`skillcellar check` asks the site which of this computer's skills and plugins their sources have moved past, as the last sync reported them, and prints each with the command that updates it: a skill `npx skills` installed globally, when the git tree of its folder on the repository's default branch is no longer the one its lock recorded (`npx skills update <name> -g`; a skill whose folder is not named as its SKILL.md names it updates with `npx skills update -g`); a plugin, when its marketplace entry declares a version other than the installed one (`claude plugin update <plugin>@<marketplace>`); a skill you wrote or bought, in a folder of its name, when that folder is an older version of it (`skillcellar pull <name> --force`, or `<username>/<name>` for one you bought, with `--scope project` in a project; it replaces the folder with the site's latest version, which is why a folder that is no version of the skill, edited here or another skill of that name, is listed as not checked instead). It then lists what the site cannot check, each with why: a skill `npx skills` installed in a project, whose lock hashes the files, which GitHub does not list; a plugin reported without a version, whose marketplace entry declares none, whose marketplace no longer lists it, or whose marketplace is not on GitHub; a repository that is private, gone or too large to list, or whose folder is no longer at the recorded path; and what the site did not read this time, past its 40 sources or 8 seconds a check or its GitHub budget, which the next check tries again. The site reads each repository and marketplace once an hour, public ones only, with its own GitHub token. `--json` prints the site's answer as it is. Sync first: the check reads what the last sync reported, as its last line says, not the computer as it is now.
|
|
29
29
|
|
|
30
|
+
## Technologies
|
|
31
|
+
|
|
32
|
+
At the first `login` or `sync` at a terminal, the CLI asks once whether to report each project's technologies, such as `nextjs` or `postgres`, so the site can show what fits each project. Until you say yes, it reads and sends nothing about them. `skillcellar tech on` and `skillcellar tech off` change the answer; `skillcellar tech` says which it is.
|
|
33
|
+
|
|
34
|
+
With technologies on, each sync downloads the site's public list at `/tech.json`, without your key, and checks each project where it reports something against it, on this computer:
|
|
35
|
+
|
|
36
|
+
- the files git tracks in the project (the files at its root, when git tracks none), up to three folders below its root, outside folders such as `node_modules`, `vendor`, `dist` or `examples`. A file in a folder starting with a dot counts only where the list names that folder, such as `.github/workflows`, and a manifest there is never read;
|
|
37
|
+
- at the project's root, the few files the list names there, such as `.neon`, even when git does not track them;
|
|
38
|
+
- the package names in its `package.json`, `composer.json`, `requirements*.txt`, `*-requirements.txt`, `requirements/*.txt`, `pyproject.toml`, `Pipfile`, `go.mod`, `Cargo.toml`, `Gemfile` and `pubspec.yaml` files, up to 40 of them, each at most 1 MiB. It never opens a lockfile, a `.env` file or a file through a link.
|
|
39
|
+
|
|
40
|
+
Only the ids of the list that match are sent, at most 32 a project. A project whose folder it cannot read, or whose manifests it cannot get through within two seconds, sends none, and the site keeps what it had there. Turned off, each sync takes them back from the site until one of the whole computer has gone through. When the site's list cannot be read, or the technologies would take a sync past its 5,000 items, the sync goes on without them, says so, and the site keeps what it had.
|
|
41
|
+
|
|
30
42
|
## Backups
|
|
31
43
|
|
|
32
44
|
To move to a new computer, or back onto one you reinstall, follow the step-by-step guide at https://skillcellar.com/guides/new-computer.
|
|
@@ -64,7 +76,8 @@ For each place, global or a project:
|
|
|
64
76
|
|
|
65
77
|
- a skill's or plugin's name, the GitHub repository or marketplace it comes from as an https address, its folder in that repository, and a hash or version;
|
|
66
78
|
- an MCP server's name, its transport (stdio, HTTP or SSE), and for stdio the command's own name, such as `npx`, with the package a runner such as `npx`, `uvx`, `bunx` or `pnpm dlx` starts; for HTTP and SSE, the address's origin and its path up to the first part that is not a short word such as `mcp`, `sse` or `v1`, since some addresses carry a key there. A flag the CLI does not know ends its search for the package, since its value could be a key; a flag written as `--flag=value` is skipped, and its value is never sent. Never its `env` or header values, its other arguments, a query, or a user name and password. WebSocket and SDK servers are left out;
|
|
67
|
-
- a memory file's path from its folder, such as `rules/common/testing.md`, and an agent's name, each with a SHA-256 of the file, never its contents
|
|
79
|
+
- a memory file's path from its folder, such as `rules/common/testing.md`, and an agent's name, each with a SHA-256 of the file, never its contents;
|
|
80
|
+
- with technologies on, a project's technologies, as ids of the site's list, never a package's name.
|
|
68
81
|
|
|
69
82
|
The project's name is the one you give with `--project`, else its git remote's `owner/repo`, else its folder's name, and a GitHub remote is also sent as the project's repository; the site keeps the name only as a slug, such as `owner-repo`, and a folder's name as you see it. It sends no path on your computer; a name it cannot send as it is, it lists and leaves out.
|
|
70
83
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "skillcellar",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Sync what Claude Code has on this computer (skills, plugins, MCP servers, memory and agents) to your SkillCellar library, by name and hash, list the tasks you queue there for Claude Code to carry out, check which skills and plugins are outdated, and apply playbooks.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"claude-code",
|
package/src/cli.mjs
CHANGED
|
@@ -15,6 +15,7 @@ import { checkProjectSkills, installPulled, PULL_REF_RE, readPulled } from "./pu
|
|
|
15
15
|
import { runRestore } from "./restore.mjs";
|
|
16
16
|
import { readJson, scanGlobal, scanProject } from "./scan.mjs";
|
|
17
17
|
import { installSkill, skillFolder, skillState } from "./skill.mjs";
|
|
18
|
+
import { detectTech, loadRules, techChoice, techServed } from "./tech.mjs";
|
|
18
19
|
|
|
19
20
|
const DEFAULT_SITE = "https://skillcellar.com";
|
|
20
21
|
const VERSION = JSON.parse(await readFile(new URL("../package.json", import.meta.url), "utf8")).version;
|
|
@@ -63,6 +64,11 @@ Usage:
|
|
|
63
64
|
missing with how to install it. It reads this computer as sync does, and the site keeps none of it.
|
|
64
65
|
skillcellar skill
|
|
65
66
|
Install or update the skillcellar skill, which tells Claude Code how to carry out your tasks.
|
|
67
|
+
skillcellar tech [on|off]
|
|
68
|
+
Turn on or off reporting each project's technologies at sync, such as nextjs or postgres:
|
|
69
|
+
this computer checks your projects against the site's public list at <site>/tech.json and
|
|
70
|
+
sends only the ids that match, never a package's name, a file or a path. Off, the next sync
|
|
71
|
+
of the whole computer removes the ones the site has. Alone, it says which it is.
|
|
66
72
|
skillcellar backup [--repo <owner>/<repo>] [--ssh] [--allow <file>] [--dry-run]
|
|
67
73
|
Push this computer's memory files, rules, agents, MCP config (every env and header value,
|
|
68
74
|
address credential and key-like argument replaced by a \${NAME} placeholder) and skills'
|
|
@@ -158,6 +164,16 @@ async function offerSkill(print) {
|
|
|
158
164
|
else print(`The skillcellar skill is not ${state === "missing" ? "installed" : "up to date"}; run skillcellar skill to ${verb.toLowerCase()} it.`);
|
|
159
165
|
}
|
|
160
166
|
|
|
167
|
+
/** The one question that turns technologies on: nothing about them leaves this computer before a yes. */
|
|
168
|
+
const techQuestion = (site) =>
|
|
169
|
+
`Also report each project's technologies, such as nextjs or postgres, so SkillCellar can show what fits each project? This computer checks your projects against the site's public list at ${new URL(site).host}/tech.json and sends only the ids that match, never a package's name, a file or a path. Change it any time with skillcellar tech on or off. [y/N] `;
|
|
170
|
+
|
|
171
|
+
const askTech = async (site) => /^y(es)?$/i.test(await ask(techQuestion(site)));
|
|
172
|
+
|
|
173
|
+
/** What turning technologies on or off just did, in one line. */
|
|
174
|
+
const techSaved = (tech, site) =>
|
|
175
|
+
tech === "on" ? `Technologies on: each sync reports your projects' technologies as ids from ${new URL(site).host}/tech.json.` : "Technologies off: run skillcellar tech on to report them.";
|
|
176
|
+
|
|
161
177
|
async function login(flags, print) {
|
|
162
178
|
const site = siteOrigin(flags.site ?? process.env.SKILLCELLAR_URL ?? DEFAULT_SITE);
|
|
163
179
|
if (!site) throw new UsageError("--site must be an https address, such as https://skillcellar.com, or http on this computer for development.");
|
|
@@ -169,10 +185,19 @@ async function login(flags, print) {
|
|
|
169
185
|
const earlier = await readConfig().catch(() => null);
|
|
170
186
|
const backup = earlier?.site === site && earlier.backup ? { backup: earlier.backup } : {};
|
|
171
187
|
const folders = Array.isArray(earlier?.folders) ? { folders: earlier.folders } : {};
|
|
172
|
-
|
|
188
|
+
// So is the answer about technologies, which is this site's to know.
|
|
189
|
+
const tech = earlier?.site === site && typeof earlier.tech === "string" ? { tech: earlier.tech } : {};
|
|
190
|
+
const saved = { site, key, device: { id: device.id, slug: device.slug, name: device.name }, ...backup, ...folders, ...tech };
|
|
191
|
+
await writeConfig(saved);
|
|
173
192
|
const who = library.account?.email ? ` as ${library.account.email}` : "";
|
|
174
193
|
print(`Signed in to ${new URL(site).host}${who}. This computer syncs as ${device.name}.`);
|
|
175
194
|
await offerSkill(print);
|
|
195
|
+
// Asked here as well: a sync Claude Code runs has no terminal to ask at.
|
|
196
|
+
if (!tech.tech && interactive()) {
|
|
197
|
+
const on = (await askTech(site)) ? "on" : "off";
|
|
198
|
+
await writeConfig({ ...saved, tech: on });
|
|
199
|
+
print(techSaved(on, site));
|
|
200
|
+
}
|
|
176
201
|
}
|
|
177
202
|
|
|
178
203
|
const plural = (n, word, words = `${word}s`) => `${n} ${n === 1 ? word : words}`;
|
|
@@ -227,6 +252,21 @@ async function sync(flags, print, warn) {
|
|
|
227
252
|
// Read from the wrong folder, a sync would report nothing and remove what the last one reported.
|
|
228
253
|
await checkClaudeFolder(claudeDir);
|
|
229
254
|
|
|
255
|
+
// Technologies go only after a yes, and only to a site that serves its list: one without it gets
|
|
256
|
+
// the payload it always had, and keeps what it holds.
|
|
257
|
+
const choice = await techChoice(config?.tech, Boolean(config) && !dryRun && interactive(), () => askTech(site));
|
|
258
|
+
if (choice.asked) {
|
|
259
|
+
await writeConfig({ ...config, tech: choice.tech });
|
|
260
|
+
print(techSaved(choice.tech, site));
|
|
261
|
+
}
|
|
262
|
+
// Taking them back needs no rules, only a site that serves the list, whatever its version.
|
|
263
|
+
const rules = choice.tech === "on" && site ? await loadRules(site, VERSION) : null;
|
|
264
|
+
const clearing = choice.tech === "clearing" && site !== null && (await techServed(site, VERSION));
|
|
265
|
+
if (site && ((choice.tech === "on" && !rules) || (choice.tech === "clearing" && !clearing))) {
|
|
266
|
+
warn(`Technologies not ${choice.tech === "on" ? "reported" : "removed"} this time: ${new URL(site).host}/tech.json could not be read. The site keeps what it had.`);
|
|
267
|
+
}
|
|
268
|
+
const reportTech = rules !== null;
|
|
269
|
+
|
|
230
270
|
const claudeJson = (await readJson(claudeJsonPath(home))) ?? {};
|
|
231
271
|
const cwd = process.cwd();
|
|
232
272
|
const whole = flags.project === undefined;
|
|
@@ -251,8 +291,9 @@ async function sync(flags, print, warn) {
|
|
|
251
291
|
continue;
|
|
252
292
|
}
|
|
253
293
|
let scanned;
|
|
294
|
+
const tracked = await trackedFiles(project.root);
|
|
254
295
|
try {
|
|
255
|
-
scanned = await scanProject({ root: project.root, folders: [...project.folders], claudeDir, claudeJson, tracked
|
|
296
|
+
scanned = await scanProject({ root: project.root, folders: [...project.folders], claudeDir, claudeJson, tracked });
|
|
256
297
|
} catch (error) {
|
|
257
298
|
// One project's broken file stops only that project in a sync of the whole computer.
|
|
258
299
|
if (!whole) throw error;
|
|
@@ -261,7 +302,15 @@ async function sync(flags, print, warn) {
|
|
|
261
302
|
continue;
|
|
262
303
|
}
|
|
263
304
|
skipped.push(...scanned.skipped);
|
|
264
|
-
|
|
305
|
+
// The site keeps a project's technologies only where the device reports something, so a place
|
|
306
|
+
// with nothing lists none; one the CLI could not read lists nothing at all, and keeps what the site had.
|
|
307
|
+
let tech = {};
|
|
308
|
+
if (reportTech) {
|
|
309
|
+
const found = scanned.items.length ? await detectTech(rules, project.root, tracked) : [];
|
|
310
|
+
if (found) tech = { tech: found };
|
|
311
|
+
else warn(`Technologies of ${project.name} not read this time. The site keeps what it had there.`);
|
|
312
|
+
}
|
|
313
|
+
const place = { scope: "project", project: { name: project.name, repo: project.repo, remote: project.remote }, items: scanned.items, ...tech };
|
|
265
314
|
// A complete sync replaces every place on the device, so a project with nothing in it needs no place.
|
|
266
315
|
if (whole && !scanned.items.length) empty.push(place);
|
|
267
316
|
else places.push(place);
|
|
@@ -282,8 +331,15 @@ async function sync(flags, print, warn) {
|
|
|
282
331
|
}
|
|
283
332
|
|
|
284
333
|
const device = config?.device.id ?? null;
|
|
285
|
-
const payload = { device, complete, places: [{ scope: "global", project: null, items: global.items }, ...places] };
|
|
286
|
-
|
|
334
|
+
const payload = { device, complete, ...(reportTech || clearing ? { tech: reportTech } : {}), places: [{ scope: "global", project: null, items: global.items }, ...places] };
|
|
335
|
+
// A project's technologies count as items on the site; past the cap with them, the sync goes without.
|
|
336
|
+
const counted = (withTech) => payload.places.reduce((total, place) => total + place.items.length + (withTech ? (place.tech?.length ?? 0) : 0), 0);
|
|
337
|
+
let items = counted(true);
|
|
338
|
+
if (items > ITEMS_PER_SYNC && counted(false) <= ITEMS_PER_SYNC) {
|
|
339
|
+
payload.places = payload.places.map((place) => Object.fromEntries(Object.entries(place).filter(([key]) => key !== "tech")));
|
|
340
|
+
items = counted(false);
|
|
341
|
+
warn(`Technologies not reported this time: with them, this sync passes the ${ITEMS_PER_SYNC} items one sync takes. The site keeps what it had.`);
|
|
342
|
+
}
|
|
287
343
|
if (items > ITEMS_PER_SYNC) {
|
|
288
344
|
throw new Error(`This computer has ${items} items, and one sync takes ${ITEMS_PER_SYNC}. Sync the projects you need from their folders with --project <name>.`);
|
|
289
345
|
}
|
|
@@ -294,6 +350,12 @@ async function sync(flags, print, warn) {
|
|
|
294
350
|
}
|
|
295
351
|
const result = await call(site, config.key, "POST", "/api/v1/sync", payload, VERSION);
|
|
296
352
|
print(summary(payload, result, config.device.name, whole));
|
|
353
|
+
if (reportTech && typeof result.tech === "number") print(`Reported the technologies of ${plural(result.tech, "project")}.`);
|
|
354
|
+
// Taken back everywhere once a sync of the whole computer has gone through.
|
|
355
|
+
if (clearing && complete) {
|
|
356
|
+
await writeConfig({ ...config, tech: "off" });
|
|
357
|
+
print(`Removed this computer's technologies from ${new URL(site).host}.`);
|
|
358
|
+
}
|
|
297
359
|
for (const note of Array.isArray(result.left) ? result.left : []) warn(`Left out ${note}`);
|
|
298
360
|
const closed = Array.isArray(result.closed) ? result.closed : [];
|
|
299
361
|
if (closed.length) print(closedLine(closed));
|
|
@@ -449,6 +511,28 @@ async function restore(flags, print, warn) {
|
|
|
449
511
|
if (added.length) await writeConfig({ ...config, folders: [...remembered, ...added] });
|
|
450
512
|
}
|
|
451
513
|
|
|
514
|
+
/**
|
|
515
|
+
* Turns reporting technologies on or off, or says which it is. Off after they were on, the site
|
|
516
|
+
* still has them until a sync of the whole computer takes them back ("clearing").
|
|
517
|
+
*/
|
|
518
|
+
async function tech(word, print) {
|
|
519
|
+
const { config, site } = await signedIn();
|
|
520
|
+
const host = new URL(site).host;
|
|
521
|
+
if (word === undefined) {
|
|
522
|
+
const state = {
|
|
523
|
+
on: `This computer reports each project's technologies at every sync, as ids from ${host}/tech.json. Run skillcellar tech off to stop.`,
|
|
524
|
+
clearing: `Technologies are off. The next sync of the whole computer removes this computer's technologies from ${host}.`,
|
|
525
|
+
off: "This computer does not report its projects' technologies. Run skillcellar tech on to start.",
|
|
526
|
+
}[config.tech];
|
|
527
|
+
print(state ?? "Technologies are not reported: you have not been asked yet. A sync at a terminal asks, or run skillcellar tech on.");
|
|
528
|
+
return;
|
|
529
|
+
}
|
|
530
|
+
if (word !== "on" && word !== "off") throw new UsageError("Run tech on or tech off, or tech alone to see which it is.");
|
|
531
|
+
const next = word === "on" ? "on" : config.tech === "on" || config.tech === "clearing" ? "clearing" : "off";
|
|
532
|
+
await writeConfig({ ...config, tech: next });
|
|
533
|
+
print(next === "clearing" ? `Technologies off: the next sync of the whole computer removes this computer's technologies from ${host}.` : techSaved(next, site));
|
|
534
|
+
}
|
|
535
|
+
|
|
452
536
|
/** Installs or updates the skillcellar skill without asking: the command itself is the yes. */
|
|
453
537
|
async function skill(print) {
|
|
454
538
|
const claudeDir = claudeFolder(homedir());
|
|
@@ -515,8 +599,8 @@ export async function main(argv, toOut = console.log, toErr = console.error) {
|
|
|
515
599
|
return flags.help ? 0 : 2;
|
|
516
600
|
}
|
|
517
601
|
try {
|
|
518
|
-
// Only pull
|
|
519
|
-
const [action, ref, ...rest] = command === "playbook" ? extra : command === "pull" ? [undefined, ...extra] : [undefined, undefined, ...extra];
|
|
602
|
+
// Only pull and tech take an argument, the skill it installs or on or off, and playbook two: what to do, and with which playbook.
|
|
603
|
+
const [action, ref, ...rest] = command === "playbook" ? extra : command === "pull" || command === "tech" ? [undefined, ...extra] : [undefined, undefined, ...extra];
|
|
520
604
|
if (rest.length) throw new UsageError(`Unexpected ${rest.join(" ")}.`);
|
|
521
605
|
if (command === "login") await login(flags, print);
|
|
522
606
|
else if (command === "sync") await sync(flags, print, warn);
|
|
@@ -525,6 +609,7 @@ export async function main(argv, toOut = console.log, toErr = console.error) {
|
|
|
525
609
|
else if (command === "pull") await pull(flags, ref, print);
|
|
526
610
|
else if (command === "playbook") await playbook(flags, action, ref, print, warn);
|
|
527
611
|
else if (command === "skill") await skill(print);
|
|
612
|
+
else if (command === "tech") await tech(ref, print);
|
|
528
613
|
else if (command === "backup") await backup(flags, print, warn);
|
|
529
614
|
else if (command === "restore") await restore(flags, print, warn);
|
|
530
615
|
else throw new UsageError(`Unknown command ${command}.`);
|
|
@@ -0,0 +1,290 @@
|
|
|
1
|
+
// Readers for the manifests that list a project's packages. Each takes the file's text and answers
|
|
2
|
+
// the names it lists, never throwing: a file it cannot read lists nothing. The names stay on this
|
|
3
|
+
// computer; only the ids of the technologies they match are sent.
|
|
4
|
+
|
|
5
|
+
const NAME_MAX = 214;
|
|
6
|
+
// Longer lines are skipped by the line readers: no dependency is written on one, and a pattern run on
|
|
7
|
+
// a long hostile line could take minutes.
|
|
8
|
+
const LINE_MAX = 1000;
|
|
9
|
+
const lines = (text) => text.split(/\r?\n/).filter((line) => line.length <= LINE_MAX);
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* The registry whose manifest a file is, by its name and its folder's, or null. Lockfiles are never
|
|
13
|
+
* read. pip's requirements go by many names: requirements.txt, dev-requirements.txt, requirements/base.txt.
|
|
14
|
+
*/
|
|
15
|
+
export function manifestEcosystem(file, folder = "") {
|
|
16
|
+
if (file === "package.json") return "npm";
|
|
17
|
+
if (file === "composer.json") return "composer";
|
|
18
|
+
if (file === "pyproject.toml" || file === "Pipfile" || /^(?:[\w.-]+[-_.])?requirements[\w.-]*\.txt$/.test(file) || (folder === "requirements" && file.endsWith(".txt"))) return "pypi";
|
|
19
|
+
if (file === "go.mod") return "go";
|
|
20
|
+
if (file === "Cargo.toml") return "cargo";
|
|
21
|
+
if (file === "Gemfile") return "gem";
|
|
22
|
+
if (file === "pubspec.yaml") return "pub";
|
|
23
|
+
return null;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/** A package name as its registry compares it: PyPI by PEP 503, crates with - for _, and npm, Composer and gems in lowercase. */
|
|
27
|
+
export function normalizeName(ecosystem, name) {
|
|
28
|
+
if (ecosystem === "pypi") return name.toLowerCase().replace(/[-_.]+/g, "-");
|
|
29
|
+
if (ecosystem === "cargo") return name.toLowerCase().replace(/_/g, "-");
|
|
30
|
+
if (ecosystem === "npm" || ecosystem === "composer" || ecosystem === "gem") return name.toLowerCase();
|
|
31
|
+
return name;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/** The package names a manifest lists, in its order, as written. */
|
|
35
|
+
export function dependencyNames(ecosystem, file, text) {
|
|
36
|
+
try {
|
|
37
|
+
const body = text.replace(/^/, "");
|
|
38
|
+
const names =
|
|
39
|
+
ecosystem === "npm"
|
|
40
|
+
? npmNames(body)
|
|
41
|
+
: ecosystem === "composer"
|
|
42
|
+
? composerNames(body)
|
|
43
|
+
: ecosystem === "pypi"
|
|
44
|
+
? file === "pyproject.toml"
|
|
45
|
+
? pyprojectNames(body)
|
|
46
|
+
: file === "Pipfile"
|
|
47
|
+
? pipfileNames(body)
|
|
48
|
+
: requirementsNames(body)
|
|
49
|
+
: ecosystem === "go"
|
|
50
|
+
? goNames(body)
|
|
51
|
+
: ecosystem === "cargo"
|
|
52
|
+
? cargoNames(body)
|
|
53
|
+
: ecosystem === "gem"
|
|
54
|
+
? gemNames(body)
|
|
55
|
+
: ecosystem === "pub"
|
|
56
|
+
? pubNames(body)
|
|
57
|
+
: [];
|
|
58
|
+
return names.filter((name) => typeof name === "string" && name !== "" && name.length <= NAME_MAX && !/\s/.test(name));
|
|
59
|
+
} catch {
|
|
60
|
+
return [];
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
const isTable = (value) => value !== null && typeof value === "object" && !Array.isArray(value);
|
|
65
|
+
|
|
66
|
+
function npmNames(text) {
|
|
67
|
+
const json = JSON.parse(text);
|
|
68
|
+
if (!isTable(json)) return [];
|
|
69
|
+
const names = [];
|
|
70
|
+
for (const list of ["dependencies", "devDependencies", "optionalDependencies", "peerDependencies"]) {
|
|
71
|
+
if (!isTable(json[list])) continue;
|
|
72
|
+
for (const [name, spec] of Object.entries(json[list])) {
|
|
73
|
+
names.push(name);
|
|
74
|
+
// "x": "npm:real@1" installs real under the name x.
|
|
75
|
+
if (typeof spec === "string" && spec.startsWith("npm:")) {
|
|
76
|
+
const alias = spec.slice(4);
|
|
77
|
+
const at = alias.indexOf("@", alias.startsWith("@") ? 1 : 0);
|
|
78
|
+
names.push(at === -1 ? alias : alias.slice(0, at));
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
return names;
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
function composerNames(text) {
|
|
86
|
+
const json = JSON.parse(text);
|
|
87
|
+
if (!isTable(json)) return [];
|
|
88
|
+
const platform = (name) => name === "php" || name.startsWith("ext-") || name.startsWith("lib-") || name.startsWith("composer-");
|
|
89
|
+
return ["require", "require-dev"].flatMap((list) => (isTable(json[list]) ? Object.keys(json[list]).filter((name) => !platform(name.toLowerCase())) : []));
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
/** A requirement's name, as PEP 508 writes one first, or null for anything else: an option, a path or a URL. */
|
|
93
|
+
function requirementName(requirement) {
|
|
94
|
+
const match = /^([A-Za-z0-9](?:[A-Za-z0-9._-]*[A-Za-z0-9])?)(?=$|[\s[;=<>!~@,(])/.exec(requirement.trim());
|
|
95
|
+
return match ? match[1] : null;
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
function requirementsNames(text) {
|
|
99
|
+
const names = [];
|
|
100
|
+
for (const line of lines(text.replace(/\\\r?\n/g, " "))) {
|
|
101
|
+
const requirement = line.replace(/(^|\s)#.*$/, "").trim();
|
|
102
|
+
if (!requirement || requirement.startsWith("-")) continue;
|
|
103
|
+
const name = requirementName(requirement);
|
|
104
|
+
if (name) names.push(name);
|
|
105
|
+
}
|
|
106
|
+
return names;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/** A line of TOML up to a # outside a string. */
|
|
110
|
+
function withoutComment(line) {
|
|
111
|
+
let quote = null;
|
|
112
|
+
for (let i = 0; i < line.length; i += 1) {
|
|
113
|
+
const char = line[i];
|
|
114
|
+
if (quote) {
|
|
115
|
+
if (char === "\\" && quote === '"') i += 1;
|
|
116
|
+
else if (char === quote) quote = null;
|
|
117
|
+
} else if (char === '"' || char === "'") quote = char;
|
|
118
|
+
else if (char === "#") return line.slice(0, i);
|
|
119
|
+
}
|
|
120
|
+
return line;
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
/** How deep in brackets and braces, outside strings, a line of a TOML value leaves it, from `depth`. */
|
|
124
|
+
function depthAfter(line, depth) {
|
|
125
|
+
let quote = null;
|
|
126
|
+
for (let i = 0; i < line.length; i += 1) {
|
|
127
|
+
const char = line[i];
|
|
128
|
+
if (quote) {
|
|
129
|
+
if (char === "\\" && quote === '"') i += 1;
|
|
130
|
+
else if (char === quote) quote = null;
|
|
131
|
+
} else if (char === '"' || char === "'") quote = char;
|
|
132
|
+
else if (char === "[" || char === "{") depth += 1;
|
|
133
|
+
else if (char === "]" || char === "}") depth -= 1;
|
|
134
|
+
}
|
|
135
|
+
return depth;
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
/** The strings in a TOML value, leaving out those inside an inline table such as { include-group = "x" }. */
|
|
139
|
+
function valueStrings(value) {
|
|
140
|
+
const strings = [];
|
|
141
|
+
let depth = 0;
|
|
142
|
+
let quote = null;
|
|
143
|
+
let start = 0;
|
|
144
|
+
for (let i = 0; i < value.length; i += 1) {
|
|
145
|
+
const char = value[i];
|
|
146
|
+
if (quote) {
|
|
147
|
+
if (char === "\\" && quote === '"') i += 1;
|
|
148
|
+
else if (char === quote) {
|
|
149
|
+
if (depth === 0) strings.push(value.slice(start, i));
|
|
150
|
+
quote = null;
|
|
151
|
+
}
|
|
152
|
+
} else if (char === '"' || char === "'") {
|
|
153
|
+
quote = char;
|
|
154
|
+
start = i + 1;
|
|
155
|
+
} else if (char === "{") depth += 1;
|
|
156
|
+
else if (char === "}") depth -= 1;
|
|
157
|
+
}
|
|
158
|
+
return strings;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/** A TOML key's name: a quoted one as it is, a dotted one by its first part. */
|
|
162
|
+
const keyName = (key) => (/^["']/.test(key) ? key.slice(1, -1) : key.split(".")[0].trim());
|
|
163
|
+
|
|
164
|
+
/**
|
|
165
|
+
* The tables and key = value lines of a TOML file, as far as the readers here need: each table's
|
|
166
|
+
* header (key null), then each of its keys with the value's text, an array or inline table spread
|
|
167
|
+
* over several lines joined.
|
|
168
|
+
*/
|
|
169
|
+
function* tomlEntries(text) {
|
|
170
|
+
const rows = lines(text);
|
|
171
|
+
let section = "";
|
|
172
|
+
for (let i = 0; i < rows.length; i += 1) {
|
|
173
|
+
const line = withoutComment(rows[i]).trim();
|
|
174
|
+
if (!line) continue;
|
|
175
|
+
if (line.startsWith("[")) {
|
|
176
|
+
const header = /^\[\[?([^\]]+)\]\]?$/.exec(line);
|
|
177
|
+
section = header ? header[1].trim() : "";
|
|
178
|
+
if (header) yield { section, key: null, value: "" };
|
|
179
|
+
continue;
|
|
180
|
+
}
|
|
181
|
+
const at = line.indexOf("=");
|
|
182
|
+
if (at < 1) continue;
|
|
183
|
+
let value = line.slice(at + 1).trim();
|
|
184
|
+
// Counted line by line, so a value left open to the end of the file costs one pass over it.
|
|
185
|
+
let depth = /^[[{]/.test(value) ? depthAfter(value, 0) : 0;
|
|
186
|
+
while (depth > 0 && i + 1 < rows.length) {
|
|
187
|
+
i += 1;
|
|
188
|
+
const next = withoutComment(rows[i]);
|
|
189
|
+
value += `\n${next}`;
|
|
190
|
+
depth = depthAfter(next, depth);
|
|
191
|
+
}
|
|
192
|
+
yield { section, key: line.slice(0, at).trim(), value };
|
|
193
|
+
}
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
const POETRY_RE = /^tool\.poetry\.(?:dependencies|dev-dependencies|group\.[^.]+\.dependencies)$/;
|
|
197
|
+
|
|
198
|
+
function pyprojectNames(text) {
|
|
199
|
+
const names = [];
|
|
200
|
+
for (const { section, key, value } of tomlEntries(text)) {
|
|
201
|
+
if (key === null) continue;
|
|
202
|
+
const lists = (section === "project" && keyName(key) === "dependencies") || section === "project.optional-dependencies" || section === "dependency-groups" || (section === "tool.uv" && keyName(key) === "dev-dependencies");
|
|
203
|
+
if (lists) {
|
|
204
|
+
// One push per name: spreading a long list into push() passes the engine's limit on arguments.
|
|
205
|
+
for (const requirement of valueStrings(value)) {
|
|
206
|
+
const name = requirementName(requirement);
|
|
207
|
+
if (name) names.push(name);
|
|
208
|
+
}
|
|
209
|
+
} else if (POETRY_RE.test(section) && keyName(key).toLowerCase() !== "python") names.push(keyName(key));
|
|
210
|
+
}
|
|
211
|
+
return names;
|
|
212
|
+
}
|
|
213
|
+
|
|
214
|
+
function pipfileNames(text) {
|
|
215
|
+
return [...tomlEntries(text)].flatMap(({ section, key }) => (key !== null && (section === "packages" || section === "dev-packages") ? [keyName(key)] : []));
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
const CARGO_TABLE_RE = /^(?:target\..+\.)?(?:dependencies|dev-dependencies|build-dependencies)$|^workspace\.dependencies$/;
|
|
219
|
+
const CARGO_NAMED_RE = /^(?:(?:target\..+\.)?(?:dependencies|dev-dependencies|build-dependencies)|workspace\.dependencies)\.([^.]+)$/;
|
|
220
|
+
|
|
221
|
+
function cargoNames(text) {
|
|
222
|
+
const names = [];
|
|
223
|
+
const quoted = (value) => /^["']([^"']+)["']$/.exec(value.trim())?.[1];
|
|
224
|
+
for (const { section, key, value } of tomlEntries(text)) {
|
|
225
|
+
if (CARGO_TABLE_RE.test(section)) {
|
|
226
|
+
if (key === null) continue;
|
|
227
|
+
names.push(keyName(key));
|
|
228
|
+
// db = { package = "rusqlite" } installs rusqlite under the name db.
|
|
229
|
+
const renamed = value.startsWith("{") ? /\bpackage\s*=\s*["']([^"']+)["']/.exec(value)?.[1] : undefined;
|
|
230
|
+
if (renamed) names.push(renamed);
|
|
231
|
+
continue;
|
|
232
|
+
}
|
|
233
|
+
const named = CARGO_NAMED_RE.exec(section);
|
|
234
|
+
if (!named) continue;
|
|
235
|
+
if (key === null) names.push(keyName(named[1]));
|
|
236
|
+
else if (keyName(key) === "package" && quoted(value)) names.push(quoted(value));
|
|
237
|
+
}
|
|
238
|
+
return names;
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
function goNames(text) {
|
|
242
|
+
const names = [];
|
|
243
|
+
let block = false;
|
|
244
|
+
for (const raw of lines(text)) {
|
|
245
|
+
const indirect = /\/\/\s*indirect\b/.test(raw);
|
|
246
|
+
const line = raw.replace(/\/\/.*$/, "").trim();
|
|
247
|
+
if (!line) continue;
|
|
248
|
+
let path = null;
|
|
249
|
+
if (block) {
|
|
250
|
+
if (line === ")") {
|
|
251
|
+
block = false;
|
|
252
|
+
continue;
|
|
253
|
+
}
|
|
254
|
+
path = line.split(/\s+/)[0];
|
|
255
|
+
} else if (/^require\s*\($/.test(line)) {
|
|
256
|
+
block = true;
|
|
257
|
+
continue;
|
|
258
|
+
} else {
|
|
259
|
+
path = /^require\s+(\S+)/.exec(line)?.[1] ?? null;
|
|
260
|
+
}
|
|
261
|
+
// A module's major version past 1 ends its path: github.com/jackc/pgx/v5.
|
|
262
|
+
if (path && !indirect) names.push(path.replace(/^"|"$/g, "").replace(/\/v\d+$/, ""));
|
|
263
|
+
}
|
|
264
|
+
return names;
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
function gemNames(text) {
|
|
268
|
+
return lines(text).flatMap((line) => /^gem[\s(]+["']([^"']+)["']/.exec(line.trimStart())?.[1] ?? []);
|
|
269
|
+
}
|
|
270
|
+
|
|
271
|
+
function pubNames(text) {
|
|
272
|
+
const names = [];
|
|
273
|
+
let section = false;
|
|
274
|
+
let indent = null;
|
|
275
|
+
for (const raw of lines(text)) {
|
|
276
|
+
const trimmed = raw.trim();
|
|
277
|
+
if (!trimmed || trimmed.startsWith("#")) continue;
|
|
278
|
+
const lead = raw.length - raw.trimStart().length;
|
|
279
|
+
if (lead === 0) {
|
|
280
|
+
section = /^(?:dependencies|dev_dependencies)\s*:/.test(raw);
|
|
281
|
+
indent = null;
|
|
282
|
+
continue;
|
|
283
|
+
}
|
|
284
|
+
if (!section) continue;
|
|
285
|
+
indent ??= lead;
|
|
286
|
+
const name = lead === indent ? /^([A-Za-z0-9_]+)\s*:/.exec(trimmed)?.[1] : undefined;
|
|
287
|
+
if (name) names.push(name);
|
|
288
|
+
}
|
|
289
|
+
return names;
|
|
290
|
+
}
|
package/src/tech.mjs
ADDED
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
import { lstat, open, readdir, realpath } from "node:fs/promises";
|
|
2
|
+
import { dirname, join, sep } from "node:path";
|
|
3
|
+
import { dependencyNames, manifestEcosystem, normalizeName } from "./manifests.mjs";
|
|
4
|
+
|
|
5
|
+
// Finds a project's technologies on this computer. The rules come from the site's public list at
|
|
6
|
+
// /tech.json; they are names and paths compared by prefix and suffix, never a pattern the code
|
|
7
|
+
// compiles, and only the ids they match leave the computer.
|
|
8
|
+
|
|
9
|
+
export const RULES_BYTES_MAX = 512 * 1024;
|
|
10
|
+
const RULES_VERSION = 1;
|
|
11
|
+
const ECOSYSTEMS = ["npm", "pypi", "go", "cargo", "gem", "composer", "pub"];
|
|
12
|
+
const ID_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
|
|
13
|
+
const ID_MAX = 32;
|
|
14
|
+
// The site's SYNC_TECH_MAX: the most one project reports.
|
|
15
|
+
const PER_PROJECT_MAX = 32;
|
|
16
|
+
const RULES_MAX = 500;
|
|
17
|
+
const PATTERNS_MAX = 200;
|
|
18
|
+
const PATTERN_MAX = 214;
|
|
19
|
+
const SEGMENTS_MAX = 3;
|
|
20
|
+
// A path counts up to this many segments deep: the manifests and markers that say what a project is sit near its root.
|
|
21
|
+
const PATH_SEGMENTS_MAX = 4;
|
|
22
|
+
const MANIFESTS_MAX = 40;
|
|
23
|
+
const MANIFEST_BYTES_MAX = 1024 * 1024;
|
|
24
|
+
const PROJECT_MS_MAX = 2000;
|
|
25
|
+
// Never counted, whatever the list says.
|
|
26
|
+
const ALWAYS_IGNORED = ["node_modules", ".git"];
|
|
27
|
+
|
|
28
|
+
/** Whether a rule's name or path is one this code compares safely: one `*` at most, no `..`, no backslash, no control character. */
|
|
29
|
+
const usable = (value) => typeof value === "string" && value !== "" && value.length <= PATTERN_MAX && value.split("*").length <= 2 && !value.includes("..") && !/[\\\s\p{Cc}]/u.test(value);
|
|
30
|
+
|
|
31
|
+
/** A package name or a prefix of one ending in `*`, normalised as its registry compares names. */
|
|
32
|
+
function namePattern(ecosystem, name) {
|
|
33
|
+
if (!usable(name)) return null;
|
|
34
|
+
const star = name.endsWith("*");
|
|
35
|
+
if (name.includes("*") && !star) return null;
|
|
36
|
+
const prefix = normalizeName(ecosystem, star ? name.slice(0, -1) : name);
|
|
37
|
+
return prefix ? { prefix, star } : null;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
/** A path of 1 to 3 segments, tied to the project's root when it starts with `/`, a segment holding at most one `*`. */
|
|
41
|
+
function filePattern(path) {
|
|
42
|
+
if (!usable(path)) return null;
|
|
43
|
+
const anchored = path.startsWith("/");
|
|
44
|
+
const segments = (anchored ? path.slice(1) : path).split("/");
|
|
45
|
+
if (segments.length > SEGMENTS_MAX || segments.some((segment) => segment === "" || segment === ".")) return null;
|
|
46
|
+
return {
|
|
47
|
+
anchored,
|
|
48
|
+
segments: segments.map((segment) => {
|
|
49
|
+
const at = segment.indexOf("*");
|
|
50
|
+
return at === -1 ? { literal: segment } : { prefix: segment.slice(0, at), suffix: segment.slice(at + 1) };
|
|
51
|
+
}),
|
|
52
|
+
};
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* The list the site serves, as rules this code can use, or null for a list of another version.
|
|
57
|
+
* It comes from the network, so every field is checked, and a rule it cannot read is left out.
|
|
58
|
+
*/
|
|
59
|
+
export function readRules(json) {
|
|
60
|
+
if (json?.version !== RULES_VERSION || !Array.isArray(json.tech)) return null;
|
|
61
|
+
const listed = Array.isArray(json.ignore) ? json.ignore.filter((name) => typeof name === "string" && name !== "" && name.length <= 64 && !name.includes("/")) : [];
|
|
62
|
+
const ignore = new Set([...ALWAYS_IGNORED, ...listed.slice(0, PATTERNS_MAX)]);
|
|
63
|
+
const tech = [];
|
|
64
|
+
const seen = new Set();
|
|
65
|
+
for (const entry of json.tech.slice(0, RULES_MAX)) {
|
|
66
|
+
if (typeof entry?.id !== "string" || entry.id.length > ID_MAX || !ID_RE.test(entry.id) || seen.has(entry.id)) continue;
|
|
67
|
+
const deps = {};
|
|
68
|
+
for (const ecosystem of ECOSYSTEMS) {
|
|
69
|
+
const names = Array.isArray(entry.deps?.[ecosystem]) ? entry.deps[ecosystem].slice(0, PATTERNS_MAX) : [];
|
|
70
|
+
const patterns = names.flatMap((name) => namePattern(ecosystem, name) ?? []);
|
|
71
|
+
if (patterns.length) deps[ecosystem] = patterns;
|
|
72
|
+
}
|
|
73
|
+
const files = (Array.isArray(entry.files) ? entry.files.slice(0, PATTERNS_MAX) : []).flatMap((path) => filePattern(path) ?? []);
|
|
74
|
+
if (!files.length && !Object.keys(deps).length) continue;
|
|
75
|
+
seen.add(entry.id);
|
|
76
|
+
tech.push({ id: entry.id, deps, files });
|
|
77
|
+
}
|
|
78
|
+
return { ignore, tech };
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/**
|
|
82
|
+
* The site's list of technologies, downloaded without the key and never through a redirect, or
|
|
83
|
+
* null when it cannot be had: missing, moved, too large, not JSON, too slow or of another version.
|
|
84
|
+
*/
|
|
85
|
+
export async function loadRules(site, version, { timeout = 10_000 } = {}) {
|
|
86
|
+
try {
|
|
87
|
+
const response = await fetch(new URL("/tech.json", site), {
|
|
88
|
+
headers: { accept: "application/json", "user-agent": `skillcellar/${version}` },
|
|
89
|
+
redirect: "error",
|
|
90
|
+
signal: AbortSignal.timeout(timeout),
|
|
91
|
+
});
|
|
92
|
+
if (!response.ok || !response.body) return null;
|
|
93
|
+
const chunks = [];
|
|
94
|
+
let size = 0;
|
|
95
|
+
for await (const chunk of response.body) {
|
|
96
|
+
size += chunk.length;
|
|
97
|
+
if (size > RULES_BYTES_MAX) return null;
|
|
98
|
+
chunks.push(chunk);
|
|
99
|
+
}
|
|
100
|
+
return readRules(JSON.parse(Buffer.concat(chunks).toString("utf8")));
|
|
101
|
+
} catch {
|
|
102
|
+
return null;
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
/**
|
|
107
|
+
* Whether the site serves /tech.json at all, whatever the list's version: it then takes `tech: false`,
|
|
108
|
+
* which needs no rules. A site without it, from before 0.8.0's server, would refuse the key.
|
|
109
|
+
*/
|
|
110
|
+
export async function techServed(site, version, { timeout = 10_000 } = {}) {
|
|
111
|
+
try {
|
|
112
|
+
const response = await fetch(new URL("/tech.json", site), {
|
|
113
|
+
headers: { accept: "application/json", "user-agent": `skillcellar/${version}` },
|
|
114
|
+
redirect: "error",
|
|
115
|
+
signal: AbortSignal.timeout(timeout),
|
|
116
|
+
});
|
|
117
|
+
await response.body?.cancel();
|
|
118
|
+
return response.ok;
|
|
119
|
+
} catch {
|
|
120
|
+
return false;
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
/**
|
|
125
|
+
* Whether this computer reports technologies: what the config says, or the answer to the one
|
|
126
|
+
* question, asked only where it can be. `asked` says the answer is new, for the config to keep.
|
|
127
|
+
*/
|
|
128
|
+
export async function techChoice(saved, canAsk, askYes) {
|
|
129
|
+
if (saved === "on" || saved === "off" || saved === "clearing") return { tech: saved, asked: false };
|
|
130
|
+
if (!canAsk) return { tech: null, asked: false };
|
|
131
|
+
return { tech: (await askYes()) ? "on" : "off", asked: true };
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
/** A folder or file name a pattern's segment matches; a name starting with a dot only where the rule names the dot. */
|
|
135
|
+
function segmentMatches(pattern, segment) {
|
|
136
|
+
const named = pattern.literal ?? pattern.prefix;
|
|
137
|
+
if (segment.startsWith(".") && !named.startsWith(".")) return false;
|
|
138
|
+
if (pattern.literal !== undefined) return segment === pattern.literal;
|
|
139
|
+
return segment.length >= pattern.prefix.length + pattern.suffix.length && segment.startsWith(pattern.prefix) && segment.endsWith(pattern.suffix);
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/** Whether a path, as its segments, is one a file rule names: the whole path for a rule tied to the root, else its end, with no dot folder before it. */
|
|
143
|
+
function pathMatches(pattern, path) {
|
|
144
|
+
const count = pattern.segments.length;
|
|
145
|
+
if (pattern.anchored ? path.length !== count : path.length < count) return false;
|
|
146
|
+
const tail = path.slice(-count);
|
|
147
|
+
if (!pattern.anchored && path.slice(0, -count).some((segment) => segment.startsWith("."))) return false;
|
|
148
|
+
return pattern.segments.every((segment, index) => segmentMatches(segment, tail[index]));
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
const nameMatches = (pattern, name) => (pattern.star ? name.startsWith(pattern.prefix) : name === pattern.prefix);
|
|
152
|
+
|
|
153
|
+
/** The project's files as segments, from the list git tracks, or the root's own files when it lists none. */
|
|
154
|
+
async function projectPaths(root, tracked, ignore) {
|
|
155
|
+
let listed = tracked?.size ? [...tracked] : [];
|
|
156
|
+
if (!listed.length) {
|
|
157
|
+
const entries = await readdir(root, { withFileTypes: true }).catch(() => []);
|
|
158
|
+
listed = entries.filter((entry) => entry.isFile()).map((entry) => entry.name);
|
|
159
|
+
}
|
|
160
|
+
return listed.map((path) => path.split("/")).filter((path) => path.length <= PATH_SEGMENTS_MAX && path.every((segment) => segment !== "" && !ignore.has(segment)));
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
/** Whether a path, its links resolved, is the project's folder or inside it: a linked parent folder can point anywhere. */
|
|
164
|
+
async function inside(base, path) {
|
|
165
|
+
const real = await realpath(path).catch(() => null);
|
|
166
|
+
return real !== null && (real === base || real.startsWith(base + sep));
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
/** A file or folder at the root, which git may not track, such as .neon; a link is not followed. */
|
|
170
|
+
async function isThere(base, path) {
|
|
171
|
+
const found = await lstat(path).catch(() => null);
|
|
172
|
+
return Boolean(found && (found.isFile() || found.isDirectory()) && (await inside(base, dirname(path))));
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
/** A manifest's text, or null when it is missing, a link or under one, not a plain file, or over 1 MiB. */
|
|
176
|
+
async function readManifest(base, path) {
|
|
177
|
+
const found = await lstat(path).catch(() => null);
|
|
178
|
+
if (!found?.isFile() || found.size > MANIFEST_BYTES_MAX || !(await inside(base, path))) return null;
|
|
179
|
+
let handle;
|
|
180
|
+
try {
|
|
181
|
+
handle = await open(path, "r");
|
|
182
|
+
const opened = await handle.stat();
|
|
183
|
+
if (!opened.isFile() || opened.size > MANIFEST_BYTES_MAX) return null;
|
|
184
|
+
return await handle.readFile("utf8");
|
|
185
|
+
} catch {
|
|
186
|
+
return null;
|
|
187
|
+
} finally {
|
|
188
|
+
await handle?.close().catch(() => {});
|
|
189
|
+
}
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
/**
|
|
193
|
+
* The ids of the technologies a project uses, as the rules find them in its files and manifests,
|
|
194
|
+
* in the rules' order, at most 32. Nothing it reads leaves this function but those ids, and it
|
|
195
|
+
* never throws. Null when it could not tell, because a read failed or took past its time: the
|
|
196
|
+
* sync then sends nothing for the project, and the site keeps what it had, rather than fewer ids.
|
|
197
|
+
*/
|
|
198
|
+
export async function detectTech(rules, root, tracked) {
|
|
199
|
+
try {
|
|
200
|
+
const started = Date.now();
|
|
201
|
+
const base = await realpath(root);
|
|
202
|
+
const paths = await projectPaths(root, tracked, rules.ignore);
|
|
203
|
+
const found = new Set();
|
|
204
|
+
for (const rule of rules.tech) {
|
|
205
|
+
if (rule.files.some((pattern) => paths.some((path) => pathMatches(pattern, path)))) found.add(rule.id);
|
|
206
|
+
}
|
|
207
|
+
// A file the rule ties to the root by its whole name is looked for on disk too: .neon and .vercel are rarely tracked.
|
|
208
|
+
for (const rule of rules.tech) {
|
|
209
|
+
if (found.has(rule.id)) continue;
|
|
210
|
+
for (const pattern of rule.files) {
|
|
211
|
+
if (pattern.anchored && pattern.segments.every((segment) => segment.literal !== undefined) && (await isThere(base, join(root, ...pattern.segments.map((segment) => segment.literal))))) {
|
|
212
|
+
found.add(rule.id);
|
|
213
|
+
break;
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
const wanted = new Set(rules.tech.flatMap((rule) => Object.keys(rule.deps)));
|
|
219
|
+
// A manifest in a dot folder belongs to a tool, such as a skill in .claude or a dev container, not the project.
|
|
220
|
+
const ecosystemOf = (path) => (path.slice(0, -1).some((segment) => segment.startsWith(".")) ? null : manifestEcosystem(path.at(-1), path.at(-2)));
|
|
221
|
+
const manifests = paths
|
|
222
|
+
.filter((path) => wanted.has(ecosystemOf(path)))
|
|
223
|
+
.sort((a, b) => a.length - b.length || (a.join("/") < b.join("/") ? -1 : 1))
|
|
224
|
+
.slice(0, MANIFESTS_MAX);
|
|
225
|
+
const names = new Map(ECOSYSTEMS.map((ecosystem) => [ecosystem, new Set()]));
|
|
226
|
+
for (const path of manifests) {
|
|
227
|
+
if (Date.now() - started > PROJECT_MS_MAX) return null;
|
|
228
|
+
const ecosystem = ecosystemOf(path);
|
|
229
|
+
const text = await readManifest(base, join(root, ...path));
|
|
230
|
+
if (text === null) continue;
|
|
231
|
+
for (const name of dependencyNames(ecosystem, path.at(-1), text)) names.get(ecosystem).add(normalizeName(ecosystem, name));
|
|
232
|
+
}
|
|
233
|
+
for (const rule of rules.tech) {
|
|
234
|
+
if (found.has(rule.id)) continue;
|
|
235
|
+
const uses = Object.entries(rule.deps).some(([ecosystem, patterns]) => [...names.get(ecosystem)].some((name) => patterns.some((pattern) => nameMatches(pattern, name))));
|
|
236
|
+
if (uses) found.add(rule.id);
|
|
237
|
+
}
|
|
238
|
+
return rules.tech.filter((rule) => found.has(rule.id)).map((rule) => rule.id).slice(0, PER_PROJECT_MAX);
|
|
239
|
+
} catch {
|
|
240
|
+
return null;
|
|
241
|
+
}
|
|
242
|
+
}
|