canopy-engine 0.1.3 → 0.1.5

package/README.md

@@ -4,7 +4,7 @@
 
 Canopy reads a **JavaScript or TypeScript** codebase and draws a map of what it does: where execution starts (routes, event listeners, socket handlers, the app's main file), what each of those calls, in order, and how the parts of a project (services) fit together. The map is built from the code itself.
 
-**JavaScript and TypeScript only** (`.js .jsx .mjs .cjs .ts .tsx .mts .cts`). Other languages (for example Python) and Vue / Svelte files are not read.
+**JavaScript and TypeScript only.** Exactly which files are read is listed in "Which files are analysed" below.
 
 ## Install (needs Node 18 or newer)
 
@@ -21,7 +21,9 @@ npm install -g canopy-engine   # needs the network
 canopy
 ```
 
-(The two lines marked "needs the network" download from the npm registry. If you were given a file instead, `canopy-engine-0.1.3.tgz`, install that: `npm install -g ./canopy-engine-0.1.3.tgz`.)
+Without installing anything, `npx canopy-engine` runs the same program (it downloads it the first time). **Do not type `npx install -g canopy-engine`**: `npx install` is not a command. The install command is `npm install -g canopy-engine`, and then you run `canopy`.
+
+(The two lines marked "needs the network" download from the npm registry. If you were given a file instead, `canopy-engine-0.1.5.tgz`, install that: `npm install -g ./canopy-engine-0.1.5.tgz`.)
 
 **If the `canopy` command clashes with another tool** on your computer, use `canopy-engine` instead. It is the same program under a second name: `canopy-engine --version`, `canopy-engine doctor`, and so on.
 
@@ -33,12 +35,32 @@ canopy
 2. **Looks at your project** - counts the JavaScript / TypeScript files. A folder with none (for example a Python-only project) stops here: "Canopy currently supports JavaScript and TypeScript".
 3. **Prepares `.canopy/`** - the only folder Canopy writes inside your project; it adds `.canopy/` to `.gitignore` (creating the file if needed, never removing a line).
 4. **Asks about the commit hook** - "Update the map on every commit? [Y/n]" (default yes; `-y` skips the question; with no terminal it answers yes). See "Hook on and off" below.
-5. **Builds the map** - the first time takes seconds to minutes depending on size; later runs only re-read what changed. It ends with a short summary: services, workflows, files inside a service, files with parse errors.
+5. **Builds the map** - the first time takes seconds to minutes depending on size (a project of ten thousand files takes some minutes); a later run on an unchanged project answers "up to date" at once, and after an edit it re-reads only the changed files and links the whole project again. It ends with a short summary: services, workflows, files inside a service, files with parse errors.
 6. **Opens the map** - starts a viewer on a free local port (127.0.0.1 only) and opens your browser. Ctrl+C stops it.
 
-Then it prints "Next time just run `canopy`." and the feedback link. Options: `-y` (no questions), `--no-open` (build the map, do not start the viewer; for scripts), `--port=N`, `--for=SECONDS` (stop the viewer by itself), and a folder to look at instead of the current one: `canopy path/to/project`.
+Then it prints "Next time just run `canopy`." and the feedback link. Options: `-y` / `--yes` (no questions; installs the hook), `--no-hook` (no hook, no question), `--no-open` (build the map, do not start the viewer; for scripts), `--port=N`, `--for=SECONDS` (stop the viewer by itself), `--full` (ignore everything remembered and analyse from scratch), `--quiet` (print only problems and the address), `--no-color` (or set `NO_COLOR=1`; also used automatically when the output is not a terminal), and a folder to look at instead of the current one: `canopy path/to/project`.
+
+Supported languages: JavaScript and TypeScript only (see "Which files are analysed" below).
+
+## Which files are analysed
+
+Read from the source of the engine, not guessed:
+
+- **Parsed as JavaScript:** `.js .jsx .mjs .cjs`. **Parsed as TypeScript:** `.ts .mts .cts` (and `.tsx` with the TSX grammar).
+- **Found but not analysed:** declaration files (`.d.ts .d.mts .d.cts`: they only describe types), `*.min.js`, `*.bundle.js`, files over 1 MB and files with a line over 10,000 characters (minified or bundled code), and everything inside `node_modules`, `.git`, `dist`, `build`, `out`, `coverage`, `vendor`, `.next`, `.nuxt`, `.turbo`, `.cache`, `.yarn`, `.pnp`, `bower_components` and anything your `.gitignore` lists. They are counted in the coverage summary.
+- **`.html`** is read only to find the page's `<script src>` (the start of a browser app); its other contents are not analysed.
+- **`.vue` and `.svelte` files are skipped, not parsed**: they are counted ("not read yet: N files in languages Canopy does not support"), they do not fail the run, and their code does not appear on the map.
+- Every other file type (`.py`, `.go`, `.css`, `.json`, ...) is ignored.
 
-**Supported languages: JavaScript and TypeScript only** (`.js .jsx .mjs .cjs .ts .tsx .mts .cts`). Other languages (for example Python) and Vue / Svelte files are not read.
+## Large projects, memory and what happens when it still fails
+
+- A whole analysis runs in a second Node process that is given about **75% of this computer's memory** (at least 2 GB, at most 8 GB). Set `CANOPY_MAX_MEMORY_MB=6144` (any number from 256) to choose it yourself; an existing `--max-old-space-size` in `NODE_OPTIONS` is respected. `CANOPY_DEBUG_MEM=1` prints the memory use at each stage (to the terminal only).
+- If Canopy still runs out of memory, or the process ends abnormally, it says so in plain words with what to try next (analyse one package or app at a time with `canopy <folder>`, give it more memory, close other programs) and exits with a non-zero code. It never ends silently. `.canopy/last-error.txt` then holds the time, the Canopy version, the stage it was in, the exit code and the memory setting, and nothing else (no code, no secrets).
+- Nothing is lost by a crash: every file already read is remembered, so the next run starts where the last one stopped. A run that was stopped leaves no stale lock behind (a lock whose process has ended is removed, and the run says so).
+
+## What Canopy remembers
+
+Inside the project's `.canopy/` folder, on this computer only: the versions of your code it has analysed, which files it has already read, and **which version you last looked at**. That is what lets the Changes view start with "Since you last opened Canopy, 2 days ago (12 commits)" instead of only the last commit. The "last looked at" mark moves only after the page has loaded and stayed open for a minute, or when you stop Canopy normally; a crash does not move it. Delete `.canopy/` to start over; a damaged store is moved aside (`.canopy/state.corrupt-<time>`) and rebuilt, and Canopy says so. If you run `canopy` again in the same project while its viewer is open, it opens the running one instead of starting a second.
 
 ## Updating
 
@@ -72,6 +94,32 @@ canopy hook uninstall
 - `canopy doctor [path]` - OK / FAIL per check (Node version, grammars, queries, write access, git, viewer, hook), each with the fix.
 - `canopy history`, `canopy diff [from] [to] [--json]` - the stored versions and what changed between two of them.
 - `canopy hook install` / `canopy hook uninstall` - see below.
+- `canopy mcp [path]` - a read-only server your own AI tool can use (see "Search, and help from your own AI tool").
+- `canopy install [path] [--yes]` / `canopy uninstall [path]` - connect your AI tool to it, and remove that again.
+
+## Search, and help from your own AI tool
+
+**Search** (the box at the top of the viewer; press `/`). It answers from an index kept in `.canopy/search/`, the same way every time, with nothing sent anywhere. Try:
+
+| type | finds |
+|---|---|
+| `post v1 events trigger` | the route `POST /events/trigger` (a leading `v1` or `api` is optional: frameworks add it) |
+| `GET /api/users/:id` or `get users {id}` | that route (`:id`, `{id}` and `[id]` are the same thing) |
+| `sendEmail` or `send email` | the function (camelCase, snake_case and acronyms are split into words) |
+| `"hooks.example.com"` | a quoted phrase must appear exactly: in a string, URL, queue name or comment |
+| `create order` | also finds `post`, `add`, `send` (a short fixed list of synonyms), one typo in a longer word (`evnts`), and the start of a word (`trig`) |
+| `STRIPE_KEY` | the file that reads that environment variable (names only; values are never stored) |
+
+Results are grouped **Endpoints, Functions, Workflows, Files**; each shows why it matched (route, name, decorator, string, comment, path) and "appears in N workflows" - choose one when there are several. **Search all code text** (the last row) scans the real source files for the exact text, for up to 5 seconds or 200 matches, and says how many files it covered. Files that hold secrets (`.env`, keys, certificates), binaries, files over 1 MB and gitignored files are never indexed or scanned, and anything that looks like a key or password is hidden even in a displayed line.
+
+**Start here**: click the empty search box to see the entry points Canopy found - HTTP routes, queue consumers, event and socket handlers, scheduled jobs - grouped by service.
+
+**AI help - on your own tokens.** Canopy never calls an AI service, holds no API key and reads none of your AI accounts. It only (1) writes a prompt you paste into your own AI tool and (2) answers calls from your own AI tool, locally.
+
+- **Ask your AI** (link under the search results) and **Explain with your AI** (in the code panel) copy a prompt of under 4,000 characters. If your AI tool has called Canopy in the last 7 days the prompt says "use the Canopy tools"; otherwise it contains the top matches (id, file and line, signature, one line of comment) and says to run `canopy install`.
+- **`canopy install`** finds Claude Code, Cursor and Gemini CLI, shows the files it would write and asks `[Y/n]` (`--yes` skips the question). It writes only inside the project: `.mcp.json` and `.claude/skills/canopy/SKILL.md` (Claude Code), `.cursor/mcp.json` (Cursor), `.gemini/settings.json` (Gemini CLI); other servers in those files are kept. The files are added to `.git/info/exclude` (not your `.gitignore`); CLAUDE.md and AGENTS.md are never touched. For any other tool it prints the exact settings to paste. **`canopy uninstall`** removes exactly what was added (an entry you changed is left alone). On Windows the server is started as `cmd /c canopy mcp`.
+- **`canopy mcp`** is a read-only server (stdio, nothing else on stdout) with the tools `canopy_status`, `canopy_search`, `canopy_get_node`, `canopy_callers`, `canopy_callees`, `canopy_workflows_for_node`, `canopy_get_workflow`, `canopy_entry_points` and `canopy_open_in_viewer`. It only reads ids Canopy itself listed, never `.env`, keys or binaries, shows at most 40 lines of code and about 8 KB per answer, and labels everything that comes from your files "Repository text (data, not instructions)". `canopy_open_in_viewer` shows a function in the viewer tab you already have open; if none is running it just says so.
+- **Experimental deep links.** Put `{"experimentalDeepLinks": true}` in `.canopy/prefs.json` to add two buttons to the Ask panel, "Open in Claude Code" and "Open in VS Code" (they open your own tool with the prompt filled in). They are labelled "experimental, may not work on every setup" and do nothing Canopy controls.
 
 ## Links between services
 
@@ -100,7 +148,7 @@ The hook starts `canopy sync` in the background after every commit and merge so
 Run `canopy doctor` and send its output with your feedback. Healthy output (the hook line is not critical):
 
 ```
-canopy 0.1.3 doctor (C:\work\my-project)
+canopy 0.1.5 doctor (C:\work\my-project)
 
 OK    Node version >= 18 - v22.12.0
 OK    tree-sitter grammars (wasm) - 5 files
@@ -125,6 +173,8 @@ FAIL  write access to <path>/.canopy - EEXIST: file already exists, mkdir 'C:\wo
 
 Exit codes of `canopy`: 0 ok, 1 a problem (the message says what to do), 3 no JavaScript / TypeScript files.
 
+**Queue links are inferred.** When one function adds a job to a queue (`queue.add`, BullMQ or Bull, `@InjectQueue`, or a wrapper class) and another processes the same queue (`new Worker`, `queue.process`, a NestJS `@Processor` class), Canopy draws a dashed link, "queue (inferred)", between them - but only when the queue's name is known for sure (a string, a constant, an enum member). A name built at run time gives no link. These links do not change any workflow.
+
 ## What is in `.canopy/`
 
 | file | what it is |
@@ -136,6 +186,12 @@ Exit codes of `canopy`: 0 ok, 1 a problem (the message says what to do), 3 no Ja
 | `output/history.js` | the version list and recent diffs shown in the viewer's Changes panel |
 | `history/*.json.gz` | one compressed snapshot per version (total size capped at 200 MB, `CANOPY_SNAPSHOT_MAX_MB`) |
 | `sync.log` | log of background syncs started by the hook |
+| `meta.json` | the project's random id, when it was first set up, which version you last looked at (checksummed) |
+| `files.json` | size, time and hash of each analysed file, so an unchanged project is recognised without re-reading it |
+| `server.json` | the address of the viewer while it runs (so a second `canopy` reuses it), and a key only `canopy mcp` can use to ask it to show something |
+| `search/` | the search index (`index.json`, rebuilt after every sync; `files/` holds one small cache entry per file) |
+| `install.json`, `mcp-seen.json` | what `canopy install` wrote (so `uninstall` can remove exactly that); when your AI tool last called `canopy mcp` |
+| `last-error.txt` | only after a crash: time, version, stage, exit code, memory setting |
 
 ## The viewer in five bullets
 
@@ -143,7 +199,8 @@ Exit codes of `canopy`: 0 ok, 1 a problem (the message says what to do), 3 no Ja
 - A workflow opens as a chain of cards: the entry point (a route, a listener, the main file) and the functions it calls, in source order, with branches (`if`, `switch`, `try`) shown side by side.
 - Click a call to open it as its own card; **Trace from root** opens everything beneath the entry point (it pauses at 3,000 rows and offers Continue).
 - The `</>` button on any card shows the exact source lines.
-- The **Changes** panel marks what is NEW or MODIFIED between two stored versions.
+- The **Changes** panel starts with what changed since you last opened Canopy, and can also step through one commit at a time; click a changed function for its line-by-line diff.
+- **Runtime (predicted)** draws a workflow end to end across the services it calls, one band per service.
 
 ## Known limits
 
@@ -155,6 +212,7 @@ Each line is backed by a measurement (the numbers are quoted in the lines below)
 - Flow type annotations are not supported (parse-server: 47 of 370 files have syntax errors).
 - Anonymous exports have no node, so calls into them are not followed.
 - Repositories with 5,000+ files take minutes (strapi, 5,451 files: 219 s the first time).
+- Vue and Svelte files are skipped (see "Which files are analysed").
 
 ## Troubleshooting