devlensio 0.4.0 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +137 -0
- package/dist/clustering/index.js +0 -1
- package/dist/graph/edges/apiFetchEdges.d.ts +8 -0
- package/dist/graph/edges/apiFetchEdges.js +370 -0
- package/dist/graph/index.js +4 -0
- package/dist/pipeline/index.js +9 -10
- package/dist/types.d.ts +1 -1
- package/package.json +1 -1
package/README.md
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# devlensio — the DevLens analysis engine
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/devlensio)
|
|
4
|
+
[](https://www.gnu.org/licenses/agpl-3.0)
|
|
5
|
+
|
|
6
|
+
The core engine behind [DevLens](https://github.com/devlensio/devlensOSS). It turns a TypeScript / JavaScript / React / Next.js / Node.js repository into a **typed code graph** — nodes (components, hooks, functions, stores, routes, files, …) joined by typed edges — scores every node by architectural importance, optionally **summarizes** each node with an LLM (technical / business / security), and exposes a **traversal/query API**.
|
|
7
|
+
|
|
8
|
+
`devlensio` is a **library + local server**. The user-facing tools — the `devlens` CLI, the MCP server, the Agent Skill, and the Web UI — live in [DevLens OSS](https://github.com/devlensio/devlensOSS) and consume this package.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
npm install devlensio
|
|
16
|
+
# or: bun add devlensio
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Requires Node 18+ (or Bun). An LLM provider key is needed only for summarization, not for structural analysis.
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## What it does (the pipeline)
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
Repo path
|
|
27
|
+
│
|
|
28
|
+
[1] Fingerprint detect language, framework, router, state manager, data layer, databases
|
|
29
|
+
[2] Filesystem scan extract routes (Next.js app/pages, Express, Fastify, Koa)
|
|
30
|
+
[3] Parse (ts-morph) walk every .ts/.tsx/.js/.jsx → nodes (typed params, return types, prop types)
|
|
31
|
+
[4] Edge detection many detectors → CALLS, IMPORTS, READS_FROM, WRITES_TO, PROP_PASS, EMITS,
|
|
32
|
+
LISTENS, WRAPPED_BY, GUARDS, HANDLES, TESTS, USES
|
|
33
|
+
[5] Scoring multi-pass importance scoring + noise filtering (no AI)
|
|
34
|
+
[6] Clustering cohesive cluster assignment
|
|
35
|
+
[7] Summarize (optional) topologically-ordered LLM summaries, checkpoint/resume, MapReduce
|
|
36
|
+
│
|
|
37
|
+
▼
|
|
38
|
+
Graph persisted to ~/.devlens → queried via the traversal API / CLI / MCP / UI
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Structural analysis is fast and deterministic; summarization is the only step that calls an LLM and reuses unchanged nodes across commits.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Public API
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
import {
|
|
49
|
+
analyzePipeline, // build the graph (nodes, edges, scores)
|
|
50
|
+
runSummarization, // generate technical/business/security summaries
|
|
51
|
+
computeClusters, // cohesive clustering
|
|
52
|
+
buildGraphIndex, // index nodes+edges for traversal
|
|
53
|
+
getBlastRadius, // upstream dependents ("what breaks if I change this")
|
|
54
|
+
getKHop, // downstream dependencies ("what this needs")
|
|
55
|
+
getSubgraph, // cohesive cluster around a seed
|
|
56
|
+
findCycles, // circular-dependency groups
|
|
57
|
+
resolveConfig, initConfig, // LLM provider config (~/.devlens/config.json)
|
|
58
|
+
storage, queue, // file-based graph storage + job queue singletons
|
|
59
|
+
} from "devlensio";
|
|
60
|
+
|
|
61
|
+
// Analyze a repo → graph
|
|
62
|
+
const result = await analyzePipeline("/path/to/repo", /* isGithubRepo */ false);
|
|
63
|
+
// result.allNodes, result.allEdges, result.nodeScores
|
|
64
|
+
|
|
65
|
+
// Query the graph
|
|
66
|
+
const index = buildGraphIndex(result.allNodes, result.allEdges);
|
|
67
|
+
const impact = getBlastRadius(index, "src/auth/login.ts::login", { radius: 2 });
|
|
68
|
+
const cycles = findCycles(result.allNodes, result.allEdges);
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Also exported: all core types (`CodeNode`, `CodeEdge`, `NodeType`, `EdgeType`, …), config helpers (`maskConfig`, `writeConfig`), pre-scan helpers (`readPackageDependencies`, `categorizeLibrary`), and `EDGE_LABELS`. See `dist/index.d.ts` for the full surface.
|
|
72
|
+
|
|
73
|
+
### Node & edge types
|
|
74
|
+
|
|
75
|
+
- **Node types:** `COMPONENT`, `HOOK`, `FUNCTION`, `STATE_STORE`, `UTILITY`, `FILE`, `ROUTE`, `TEST`, `STORY`, `THIRD_PARTY` (+ internal `GHOST`).
|
|
76
|
+
- **Edge types:** `CALLS`, `IMPORTS`, `READS_FROM`, `WRITES_TO`, `PROP_PASS`, `EMITS`, `LISTENS`, `WRAPPED_BY`, `GUARDS`, `HANDLES`, `TESTS`, `USES`.
|
|
77
|
+
- Each node carries an importance score and (after summarization) a technical summary, a business summary, and a security assessment (`none|low|medium|high` + notes).
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## Configuration
|
|
82
|
+
|
|
83
|
+
Summarization config lives in `~/.devlens/config.json` (set via `initConfig`/`writeConfig`, or env vars loaded with dotenv). Supported providers: **Anthropic**, **OpenAI**, **OpenRouter**, **Gemini**, **Ollama** (local).
|
|
84
|
+
|
|
85
|
+
```env
|
|
86
|
+
LLM_PROVIDER=openrouter # ollama | openai | anthropic | openrouter | gemini
|
|
87
|
+
LLM_MODEL=grok-4.1-fast
|
|
88
|
+
LLM_API_KEY=your_api_key # not needed for ollama
|
|
89
|
+
LLM_BASE_URL= # e.g. http://localhost:11434 for Ollama
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Graphs and config are stored under `~/.devlens` and shared with all DevLens tools.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## Repo layout
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
src/
|
|
100
|
+
├── fingerprint/ # detect framework, language, router, state, data layer, databases
|
|
101
|
+
├── filesystem/ # route detection (Next.js app/pages, Express, Fastify, Koa)
|
|
102
|
+
├── parser/ # ts-morph AST extraction → nodes
|
|
103
|
+
├── graph/ # edge detectors, traversal API, third-party libs, lookup maps
|
|
104
|
+
├── scoring/ # multi-pass importance scoring + noise filtering
|
|
105
|
+
├── clustering/ # cohesive cluster computation
|
|
106
|
+
├── summarizer/ # LLM summarization (technical/business/security), prompts, checkpoints
|
|
107
|
+
├── pipeline/ # analyzePipeline — orchestrates the whole analysis
|
|
108
|
+
├── jobs/ # job queue, concurrency, SSE progress events
|
|
109
|
+
├── storage/ # file-based graph persistence (~/.devlens)
|
|
110
|
+
├── config/ # provider config resolution
|
|
111
|
+
├── server/ # HTTP API server (consumed by the DevLens Web UI)
|
|
112
|
+
└── debug/ # exportGraph and dev utilities
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## Scripts
|
|
118
|
+
|
|
119
|
+
| Script | Does |
|
|
120
|
+
| :-- | :-- |
|
|
121
|
+
| `bun run dev` | watch-mode HTTP server (`src/server/index.ts`) |
|
|
122
|
+
| `bun run start` | run the HTTP server |
|
|
123
|
+
| `bun run build` | `tsc --project tsconfig.build.json` → `dist/` (the published artifact) |
|
|
124
|
+
| `bun test` | run the test suite |
|
|
125
|
+
| `bun run export-graph` | dump a graph for debugging |
|
|
126
|
+
|
|
127
|
+
---
|
|
128
|
+
|
|
129
|
+
## Relationship to DevLens OSS
|
|
130
|
+
|
|
131
|
+
`devlensio` is published to npm and consumed by [DevLens OSS](https://github.com/devlensio/devlensOSS), which provides the `devlens` CLI (`@devlensio/cli`), the MCP server, the `/devlens` Agent Skill, and the Web UI on top of this engine. The CLI binaries bundle whatever version of `devlensio` resolves at build time, so engine fixes ship to users after a `devlensio` release **and** a bump of the dependency pin in DevLens OSS.
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## License
|
|
136
|
+
|
|
137
|
+
[GNU Affero General Public License v3.0](LICENSE). If you run a modified version as a hosted service, you must release your modifications under the same license.
|
package/dist/clustering/index.js
CHANGED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import { CodeEdge, CodeNode } from "../../types.js";
|
|
2
|
+
export interface ApiFetchCall {
|
|
3
|
+
callerType: string;
|
|
4
|
+
rawUrl: string;
|
|
5
|
+
resolvedUrl: string;
|
|
6
|
+
method: string;
|
|
7
|
+
}
|
|
8
|
+
export declare function detectNextjsApiCallEdges(nodes: CodeNode[], repoPath: string): CodeEdge[];
|
|
@@ -0,0 +1,370 @@
|
|
|
1
|
+
// This file detects the backend routes of the NEXTjs only
|
|
2
|
+
import { Project, SyntaxKind } from "ts-morph";
|
|
3
|
+
import path from "node:path";
|
|
4
|
+
const CALLER_CONFIG = {
|
|
5
|
+
"fetch": { inferMethod: "from-options", defaultMethod: "GET" },
|
|
6
|
+
"axios.get": { inferMethod: "fixed", method: "GET" },
|
|
7
|
+
"axios.post": { inferMethod: "fixed", method: "POST" },
|
|
8
|
+
"axios.put": { inferMethod: "fixed", method: "PUT" },
|
|
9
|
+
"axios.delete": { inferMethod: "fixed", method: "DELETE" },
|
|
10
|
+
"axios.patch": { inferMethod: "fixed", method: "PATCH" },
|
|
11
|
+
// axios(...) called as a function. Default verb is GET (matching axios),
|
|
12
|
+
// and the first arg may be either a URL string or a config object — both
|
|
13
|
+
// handled in extractApiCallsFromFile.
|
|
14
|
+
"axios": { inferMethod: "from-options", defaultMethod: "GET" },
|
|
15
|
+
// useSWR(key, fetcher) / useSWRMutation(key, fetcher): the first arg IS the
|
|
16
|
+
// key/URL, so URL extraction works. useQuery/useSuspenseQuery/useMutation
|
|
17
|
+
// are intentionally omitted — their first arg is a query-key array or an
|
|
18
|
+
// options object, never a URL. The real request they fire is the inner
|
|
19
|
+
// fetch/axios call, which is captured on its own as a CallExpression.
|
|
20
|
+
"useSWR": { inferMethod: "fixed", method: "GET" },
|
|
21
|
+
"useSWRMutation": { inferMethod: "fixed", method: "UNKNOWN" },
|
|
22
|
+
};
|
|
23
|
+
function urlPathtoRegex(urlPath) {
|
|
24
|
+
const pattern = urlPath.split("/").map(segment => {
|
|
25
|
+
if (segment.startsWith(":") && segment.endsWith("*"))
|
|
26
|
+
return ".+"; // catch-all :slug*
|
|
27
|
+
if (segment.startsWith(":"))
|
|
28
|
+
return "[^/]+"; // dynamic :id
|
|
29
|
+
return segment.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); // escape static segments
|
|
30
|
+
}).join("\\/");
|
|
31
|
+
return new RegExp(`^${pattern}$`);
|
|
32
|
+
}
|
|
33
|
+
// Groups Next.js API ROUTE nodes by HTTP method for efficient lookup.
|
|
34
|
+
// Each route node carries a concrete method, so it is indexed under that one
|
|
35
|
+
// method key. Callers with an UNKNOWN method (e.g. useSWRMutation) are handled
|
|
36
|
+
// on the query side in matchRouteEntries by scanning every method bucket.
|
|
37
|
+
function buildRouteIndex(nodes) {
|
|
38
|
+
const index = new Map();
|
|
39
|
+
const apiRouteNodes = nodes.filter(n => n.type === "ROUTE" &&
|
|
40
|
+
n.metadata.routeKind === "nextjs" &&
|
|
41
|
+
n.metadata.routeNodeType === "API_ROUTE" &&
|
|
42
|
+
typeof n.metadata.httpMethod === "string" &&
|
|
43
|
+
typeof n.metadata.urlPath === "string");
|
|
44
|
+
for (const routeNode of apiRouteNodes) {
|
|
45
|
+
const httpMethod = routeNode.metadata.httpMethod.toUpperCase();
|
|
46
|
+
const urlPath = routeNode.metadata.urlPath;
|
|
47
|
+
const isDynamic = routeNode.metadata.isDynamic;
|
|
48
|
+
const entry = {
|
|
49
|
+
routeNode, urlPath, httpMethod, isDynamic, urlRegex: urlPathtoRegex(urlPath),
|
|
50
|
+
};
|
|
51
|
+
if (!index.has(httpMethod))
|
|
52
|
+
index.set(httpMethod, []);
|
|
53
|
+
index.get(httpMethod).push(entry);
|
|
54
|
+
}
|
|
55
|
+
return index;
|
|
56
|
+
}
|
|
57
|
+
/*
|
|
58
|
+
Extracts string value from an initializer node.
|
|
59
|
+
Handles 3 cases:
|
|
60
|
+
'string literal' → returns value directly
|
|
61
|
+
`no-substitution template` → returns value directly
|
|
62
|
+
`template ${expr} literal` → preserves ${...} as-is so normalizeUrl handles it later
|
|
63
|
+
*/
|
|
64
|
+
function resolveVariableValue(initializer) {
|
|
65
|
+
const kind = initializer.getKind();
|
|
66
|
+
if (kind === SyntaxKind.StringLiteral) {
|
|
67
|
+
return initializer.getLiteralText();
|
|
68
|
+
}
|
|
69
|
+
if (kind === SyntaxKind.NoSubstitutionTemplateLiteral) {
|
|
70
|
+
return initializer.getLiteralText();
|
|
71
|
+
}
|
|
72
|
+
if (kind === SyntaxKind.TemplateExpression) {
|
|
73
|
+
const head = initializer.getHead().getLiteralText();
|
|
74
|
+
const spans = initializer.getTemplateSpans().map((span) => `\${${span.getExpression().getText()}}${span.getLiteral().getLiteralText()}`);
|
|
75
|
+
return head + spans.join("");
|
|
76
|
+
}
|
|
77
|
+
return null; // object, function, computed — can't resolve
|
|
78
|
+
}
|
|
79
|
+
/*
|
|
80
|
+
Resolves a variable name to its string value.
|
|
81
|
+
Strategy:
|
|
82
|
+
1. Look in the same file
|
|
83
|
+
2. Walk named imports → find the source file → look there
|
|
84
|
+
Files not yet in the project are added lazily so we don't pre-load everything.
|
|
85
|
+
*/
|
|
86
|
+
function resolveUrlVariable(varName, sourceFile, project) {
|
|
87
|
+
//case 1 -> Variable exists in same file
|
|
88
|
+
const localDecl = sourceFile.getVariableDeclaration(varName);
|
|
89
|
+
if (localDecl) {
|
|
90
|
+
const init = localDecl.getInitializer();
|
|
91
|
+
if (init)
|
|
92
|
+
return resolveVariableValue(init);
|
|
93
|
+
}
|
|
94
|
+
//case 2 -> named imports
|
|
95
|
+
for (const importDecl of sourceFile.getImportDeclarations()) {
|
|
96
|
+
const match = importDecl.getNamedImports().find(n => n.getName() === varName);
|
|
97
|
+
if (!match)
|
|
98
|
+
continue;
|
|
99
|
+
// Try to get the source file — it may not be in the project yet
|
|
100
|
+
let importedFile = importDecl.getModuleSpecifierSourceFile();
|
|
101
|
+
if (!importedFile) {
|
|
102
|
+
// Lazily add the file to the project
|
|
103
|
+
const specifier = importDecl.getModuleSpecifierValue();
|
|
104
|
+
const currentDir = path.dirname(sourceFile.getFilePath());
|
|
105
|
+
const base = path.resolve(currentDir, specifier);
|
|
106
|
+
for (const ext of [".ts", ".tsx", ".js", ".jsx"]) {
|
|
107
|
+
try {
|
|
108
|
+
importedFile = project.addSourceFileAtPath(base + ext);
|
|
109
|
+
break;
|
|
110
|
+
}
|
|
111
|
+
catch {
|
|
112
|
+
continue;
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
if (!importedFile)
|
|
117
|
+
continue;
|
|
118
|
+
const importedDecl = importedFile.getVariableDeclaration(varName);
|
|
119
|
+
if (importedDecl) {
|
|
120
|
+
const init = importedDecl.getInitializer();
|
|
121
|
+
if (init)
|
|
122
|
+
return resolveVariableValue(init);
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
return null;
|
|
126
|
+
}
|
|
127
|
+
// URL Normalization
|
|
128
|
+
// Converts a raw URL string from the call site into a normalized form
|
|
129
|
+
// that can be compared against route index entries.
|
|
130
|
+
// Examples:
|
|
131
|
+
// `/api/users/${id}` → /api/users/:dynamic
|
|
132
|
+
// `/api/users/${org}/${id}` → /api/users/:dynamic/:dynamic
|
|
133
|
+
// /api/users?foo=bar → /api/users
|
|
134
|
+
// /api/users/ → /api/users
|
|
135
|
+
function normalizeUrl(rawUrl) {
|
|
136
|
+
// Skip external URLs
|
|
137
|
+
if (rawUrl.startsWith("http://") || rawUrl.startsWith("https://"))
|
|
138
|
+
return null;
|
|
139
|
+
// Must start with /
|
|
140
|
+
if (!rawUrl.startsWith("/"))
|
|
141
|
+
return null;
|
|
142
|
+
return rawUrl
|
|
143
|
+
.split("?")[0] // strip query string
|
|
144
|
+
.replace(/\$\{[^}]+\}/g, ":dynamic") // ${anything} → :dynamic
|
|
145
|
+
.replace(/\/+$/, "") // strip trailing slash
|
|
146
|
+
|| "/"; // fallback to root if empty
|
|
147
|
+
}
|
|
148
|
+
// Route Matching
|
|
149
|
+
// Matches a normalized URL + HTTP method against the route index.
|
|
150
|
+
// Rules:
|
|
151
|
+
// 1. Static routes (isDynamic=false) are matched first using === equality
|
|
152
|
+
// 2. Dynamic routes are only tried if no static match found
|
|
153
|
+
// 3. UNKNOWN method (useSWRMutation etc.) tries all entries across all methods
|
|
154
|
+
function matchRouteEntries(normalizedUrl, method, routeIndex) {
|
|
155
|
+
// Gather candidate entries for this method
|
|
156
|
+
const candidates = method === "UNKNOWN" ? [...routeIndex.values()].flat() : routeIndex.get(method) ?? [];
|
|
157
|
+
if (candidates.length === 0)
|
|
158
|
+
return [];
|
|
159
|
+
// Pass 1: static exact match
|
|
160
|
+
const staticMatches = candidates.filter(e => !e.isDynamic && e.urlPath === normalizedUrl);
|
|
161
|
+
if (staticMatches.length > 0)
|
|
162
|
+
return staticMatches;
|
|
163
|
+
// Pass 2: dynamic regex match
|
|
164
|
+
// normalizeUrl has already turned every interpolated segment into the
|
|
165
|
+
// literal token ":dynamic", which the route regex's [^/]+ (or .+ for
|
|
166
|
+
// catch-alls) matches directly — so testing the normalized URL is enough.
|
|
167
|
+
// e.g. "/api/users/:dynamic" matches the regex for route "/api/users/:id".
|
|
168
|
+
return candidates.filter(e => e.isDynamic && e.urlRegex.test(normalizedUrl));
|
|
169
|
+
}
|
|
170
|
+
// URL Argument Extraction
|
|
171
|
+
// Shared helper — reconstructs a template literal preserving ${...} as-is
|
|
172
|
+
// so normalizeUrl can replace them with :dynamic later.
|
|
173
|
+
// Same logic as resolveVariableValue's TemplateExpression branch.
|
|
174
|
+
function reconstructTemplate(templateExpr) {
|
|
175
|
+
const head = templateExpr.getHead().getLiteralText();
|
|
176
|
+
const spans = templateExpr.getTemplateSpans().map((span) => `\${${span.getExpression().getText()}}${span.getLiteral().getLiteralText()}`);
|
|
177
|
+
return head + spans.join("");
|
|
178
|
+
}
|
|
179
|
+
// Extracts a URL string from a call argument.
|
|
180
|
+
// Handles 4 cases:
|
|
181
|
+
// '/api/users' → string literal → return directly
|
|
182
|
+
// `/api/users` → no-sub template → return directly
|
|
183
|
+
// `/api/users/${id}` → template expression → preserve ${...}
|
|
184
|
+
// API_URL → identifier → resolve via resolveUrlVariable
|
|
185
|
+
// Returns null if the arg is an object / array / expression we can't resolve.
|
|
186
|
+
function extractUrlFromArg(arg, sourceFile, project) {
|
|
187
|
+
const kind = arg.getKind();
|
|
188
|
+
if (kind === SyntaxKind.StringLiteral ||
|
|
189
|
+
kind === SyntaxKind.NoSubstitutionTemplateLiteral) {
|
|
190
|
+
const url = arg.getLiteralText();
|
|
191
|
+
return { rawUrl: url, resolvedUrl: url };
|
|
192
|
+
}
|
|
193
|
+
if (kind === SyntaxKind.TemplateExpression) {
|
|
194
|
+
const url = reconstructTemplate(arg);
|
|
195
|
+
return { rawUrl: url, resolvedUrl: url };
|
|
196
|
+
}
|
|
197
|
+
if (kind === SyntaxKind.Identifier) {
|
|
198
|
+
const varName = arg.getText();
|
|
199
|
+
const resolved = resolveUrlVariable(varName, sourceFile, project);
|
|
200
|
+
if (!resolved)
|
|
201
|
+
return null;
|
|
202
|
+
return { rawUrl: varName, resolvedUrl: resolved };
|
|
203
|
+
}
|
|
204
|
+
return null;
|
|
205
|
+
}
|
|
206
|
+
// Scans EVERY call expression in a source file exactly once.
|
|
207
|
+
// For each call matching CALLER_CONFIG:
|
|
208
|
+
// - infers the HTTP method
|
|
209
|
+
// - extracts and resolves the URL argument
|
|
210
|
+
// The expensive AST traversal happens once per file here; callers then
|
|
211
|
+
// attribute the resulting calls to nodes by line range (see detectNextjsApiCallEdges).
|
|
212
|
+
// Note: useQuery/useMutation inner fetch/axios calls are captured naturally
|
|
213
|
+
// since they are also CallExpressions in the file.
|
|
214
|
+
function extractApiCallsFromFile(sourceFile, project) {
|
|
215
|
+
const results = [];
|
|
216
|
+
for (const call of sourceFile.getDescendantsOfKind(SyntaxKind.CallExpression)) {
|
|
217
|
+
const callerType = call.getExpression().getText();
|
|
218
|
+
const config = CALLER_CONFIG[callerType];
|
|
219
|
+
if (!config)
|
|
220
|
+
continue;
|
|
221
|
+
const args = call.getArguments();
|
|
222
|
+
if (args.length === 0)
|
|
223
|
+
continue;
|
|
224
|
+
// Infer HTTP method + extract URL
|
|
225
|
+
let method;
|
|
226
|
+
let urlResult;
|
|
227
|
+
if (callerType === "axios" &&
|
|
228
|
+
args[0].getKind() === SyntaxKind.ObjectLiteralExpression) {
|
|
229
|
+
// axios({ url, method }) — both url and method live in the config object.
|
|
230
|
+
// axios defaults to GET when no method is given.
|
|
231
|
+
const cfg = args[0];
|
|
232
|
+
const urlInit = cfg.getProperty("url")?.getInitializer?.();
|
|
233
|
+
urlResult = urlInit ? extractUrlFromArg(urlInit, sourceFile, project) : null;
|
|
234
|
+
method = "GET";
|
|
235
|
+
const methodInit = cfg.getProperty("method")?.getInitializer?.();
|
|
236
|
+
if (methodInit && methodInit.getKind() === SyntaxKind.StringLiteral) {
|
|
237
|
+
method = methodInit.getLiteralText().toUpperCase();
|
|
238
|
+
}
|
|
239
|
+
}
|
|
240
|
+
else if (config.inferMethod === "fixed") {
|
|
241
|
+
method = config.method;
|
|
242
|
+
urlResult = extractUrlFromArg(args[0], sourceFile, project);
|
|
243
|
+
}
|
|
244
|
+
else {
|
|
245
|
+
method = config.defaultMethod;
|
|
246
|
+
// fetch('/api', { method: 'POST' }) / axios('/api', { method }) — check 2nd arg
|
|
247
|
+
if (args.length >= 2) {
|
|
248
|
+
const optsArg = args[1];
|
|
249
|
+
if (optsArg.getKind() === SyntaxKind.ObjectLiteralExpression) {
|
|
250
|
+
const methodProp = optsArg.getProperty("method");
|
|
251
|
+
if (methodProp) {
|
|
252
|
+
const init = methodProp.getInitializer?.();
|
|
253
|
+
if (init && init.getKind() === SyntaxKind.StringLiteral) {
|
|
254
|
+
method = init.getLiteralText().toUpperCase();
|
|
255
|
+
}
|
|
256
|
+
}
|
|
257
|
+
}
|
|
258
|
+
}
|
|
259
|
+
urlResult = extractUrlFromArg(args[0], sourceFile, project);
|
|
260
|
+
}
|
|
261
|
+
if (!urlResult)
|
|
262
|
+
continue;
|
|
263
|
+
results.push({
|
|
264
|
+
startLine: call.getStartLineNumber(),
|
|
265
|
+
call: {
|
|
266
|
+
callerType,
|
|
267
|
+
rawUrl: urlResult.rawUrl,
|
|
268
|
+
resolvedUrl: urlResult.resolvedUrl,
|
|
269
|
+
method,
|
|
270
|
+
},
|
|
271
|
+
});
|
|
272
|
+
}
|
|
273
|
+
return results;
|
|
274
|
+
}
|
|
275
|
+
//MAIN FUNCTION
|
|
276
|
+
export function detectNextjsApiCallEdges(nodes, repoPath) {
|
|
277
|
+
const edges = [];
|
|
278
|
+
const dedupSet = new Set();
|
|
279
|
+
// Build route index — bail early if no Next.js API routes exist in this repo
|
|
280
|
+
const routeIndex = buildRouteIndex(nodes);
|
|
281
|
+
if (routeIndex.size === 0)
|
|
282
|
+
return [];
|
|
283
|
+
// One shared project instance — source files are added lazily per node
|
|
284
|
+
// so we never load the entire repo into ts-morph upfront
|
|
285
|
+
const project = new Project({
|
|
286
|
+
compilerOptions: {
|
|
287
|
+
allowJs: true,
|
|
288
|
+
checkJs: false,
|
|
289
|
+
strict: false,
|
|
290
|
+
},
|
|
291
|
+
skipAddingFilesFromTsConfig: true,
|
|
292
|
+
});
|
|
293
|
+
// Only scan nodes that:
|
|
294
|
+
// - are not ROUTE or THIRD_PARTY nodes themselves
|
|
295
|
+
// - have rawCode (we need the AST, not just metadata)
|
|
296
|
+
const candidateNodes = nodes.filter(n => n.type !== "ROUTE" &&
|
|
297
|
+
n.type !== "THIRD_PARTY" &&
|
|
298
|
+
typeof n.rawCode === "string" &&
|
|
299
|
+
n.rawCode.length > 0);
|
|
300
|
+
// Group candidate nodes by absolute file path so each file is parsed and
|
|
301
|
+
// its AST traversed exactly once — previously every node re-scanned the
|
|
302
|
+
// entire file's call expressions (O(calls × nodes per file)).
|
|
303
|
+
const nodesByFile = new Map();
|
|
304
|
+
for (const node of candidateNodes) {
|
|
305
|
+
const absolutePath = path.resolve(repoPath, node.filePath);
|
|
306
|
+
if (!nodesByFile.has(absolutePath))
|
|
307
|
+
nodesByFile.set(absolutePath, []);
|
|
308
|
+
nodesByFile.get(absolutePath).push(node);
|
|
309
|
+
}
|
|
310
|
+
for (const [absolutePath, fileNodes] of nodesByFile) {
|
|
311
|
+
// Reuse already-loaded file if present, otherwise add it lazily
|
|
312
|
+
let sourceFile = project.getSourceFile(absolutePath);
|
|
313
|
+
if (!sourceFile) {
|
|
314
|
+
try {
|
|
315
|
+
sourceFile = project.addSourceFileAtPath(absolutePath);
|
|
316
|
+
}
|
|
317
|
+
catch {
|
|
318
|
+
continue; // file missing, non-parseable — skip silently
|
|
319
|
+
}
|
|
320
|
+
}
|
|
321
|
+
// Scan the whole file's API calls once, then attribute by line range.
|
|
322
|
+
const locatedCalls = extractApiCallsFromFile(sourceFile, project);
|
|
323
|
+
if (locatedCalls.length === 0)
|
|
324
|
+
continue;
|
|
325
|
+
for (const located of locatedCalls) {
|
|
326
|
+
// Attribute each call to the single innermost node whose line range
|
|
327
|
+
// contains it. Nodes can nest (e.g. a component and an inner function
|
|
328
|
+
// both span the call); without picking the smallest containing range a
|
|
329
|
+
// call would be counted once per containing node, producing duplicate
|
|
330
|
+
// edges with different `from` nodes.
|
|
331
|
+
let owner = null;
|
|
332
|
+
for (const node of fileNodes) {
|
|
333
|
+
if (located.startLine < node.startLine || located.startLine > node.endLine)
|
|
334
|
+
continue;
|
|
335
|
+
if (owner === null || (node.endLine - node.startLine) < (owner.endLine - owner.startLine)) {
|
|
336
|
+
owner = node;
|
|
337
|
+
}
|
|
338
|
+
}
|
|
339
|
+
if (owner === null)
|
|
340
|
+
continue;
|
|
341
|
+
const call = located.call;
|
|
342
|
+
const normalizedUrl = normalizeUrl(call.resolvedUrl);
|
|
343
|
+
if (!normalizedUrl)
|
|
344
|
+
continue;
|
|
345
|
+
const matches = matchRouteEntries(normalizedUrl, call.method, routeIndex);
|
|
346
|
+
if (matches.length === 0)
|
|
347
|
+
continue;
|
|
348
|
+
for (const match of matches) {
|
|
349
|
+
// One edge per unique (caller node, route node, method) combination
|
|
350
|
+
const dedupKey = `${owner.id}→${match.routeNode.id}:${call.method}`;
|
|
351
|
+
if (dedupSet.has(dedupKey))
|
|
352
|
+
continue;
|
|
353
|
+
dedupSet.add(dedupKey);
|
|
354
|
+
edges.push({
|
|
355
|
+
from: owner.id,
|
|
356
|
+
to: match.routeNode.id,
|
|
357
|
+
type: "NEXTJS_API_CALL",
|
|
358
|
+
metadata: {
|
|
359
|
+
url: call.resolvedUrl,
|
|
360
|
+
rawUrl: call.rawUrl,
|
|
361
|
+
method: call.method,
|
|
362
|
+
callerType: call.callerType,
|
|
363
|
+
matchType: match.isDynamic ? "dynamic" : "exact",
|
|
364
|
+
},
|
|
365
|
+
});
|
|
366
|
+
}
|
|
367
|
+
}
|
|
368
|
+
}
|
|
369
|
+
return edges;
|
|
370
|
+
}
|
package/dist/graph/index.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { buildLookupMaps } from "./buildLookup.js";
|
|
2
|
+
import { detectNextjsApiCallEdges } from "./edges/apiFetchEdges.js";
|
|
2
3
|
import { detectCallEdges } from "./edges/callEdges.js";
|
|
3
4
|
import { detectEventEdges } from "./edges/eventEdges.js";
|
|
4
5
|
import { detectGuardEdges } from "./edges/guardEdges.js";
|
|
@@ -27,6 +28,7 @@ export function detectEdges(nodes, routeNodes, repoPath, fingerprint) {
|
|
|
27
28
|
// GUARDS — middleware to route protection
|
|
28
29
|
const guardEdges = detectGuardEdges(nodes, lookupMp, routeNodes, repoPath, fingerprint);
|
|
29
30
|
const testEdges = detectTestEdges(lookupMp, repoPath); // This does not needs nodes, as it detect edges from the file
|
|
31
|
+
const nextjsApiCallEdges = detectNextjsApiCallEdges(nodes, repoPath);
|
|
30
32
|
// Collect all dynamically-created third-party method nodes (dedup by id)
|
|
31
33
|
const newThirdPartyNodesMap = new Map();
|
|
32
34
|
for (const n of [...importResult.thirdPartyMethodNodes, ...callResult.newThirdPartyNodes]) {
|
|
@@ -46,6 +48,7 @@ export function detectEdges(nodes, routeNodes, repoPath, fingerprint) {
|
|
|
46
48
|
console.log(` TEST edges: ${testEdges.length}`);
|
|
47
49
|
console.log(` Ghost nodes created: ${eventResults.ghostNodes.length}`);
|
|
48
50
|
console.log(` Third-party method nodes: ${newThirdPartyNodes.length}`);
|
|
51
|
+
console.log(` NEXTJS_API_CALL edges: ${nextjsApiCallEdges.length}`);
|
|
49
52
|
const allEdges = [
|
|
50
53
|
...callEdges,
|
|
51
54
|
...importEdges,
|
|
@@ -56,6 +59,7 @@ export function detectEdges(nodes, routeNodes, repoPath, fingerprint) {
|
|
|
56
59
|
...routeEdges,
|
|
57
60
|
...guardEdges,
|
|
58
61
|
...testEdges,
|
|
62
|
+
...nextjsApiCallEdges,
|
|
59
63
|
];
|
|
60
64
|
console.log(`Total edges detected: ${allEdges.length}`);
|
|
61
65
|
return {
|
package/dist/pipeline/index.js
CHANGED
|
@@ -7,7 +7,7 @@ import { parseRepo } from "../parser/index.js";
|
|
|
7
7
|
import { detectEdges } from "../graph/index.js";
|
|
8
8
|
import { buildThirdPartyNodes } from "../graph/thirdPartyLibs.js";
|
|
9
9
|
import { scoreAndFilter } from "../scoring/index.js";
|
|
10
|
-
//
|
|
10
|
+
// Helpers
|
|
11
11
|
// Deterministic graphId — same repo always produces same id
|
|
12
12
|
// This ensures multiple analyses of the same repo go into the same folder
|
|
13
13
|
function generateGraphId(repoPath, isGithubRepo) {
|
|
@@ -162,7 +162,7 @@ function routesToCodeNodes(routes, repoPath) {
|
|
|
162
162
|
}
|
|
163
163
|
return nodes;
|
|
164
164
|
}
|
|
165
|
-
//
|
|
165
|
+
// analyzePipeline
|
|
166
166
|
export async function analyzePipeline(repoPath, isGithubRepo, options) {
|
|
167
167
|
const absoluteRepoPath = path.resolve(repoPath);
|
|
168
168
|
const graphId = generateGraphId(repoPath, isGithubRepo); // stable, deterministic ID based on repo path
|
|
@@ -172,11 +172,11 @@ export async function analyzePipeline(repoPath, isGithubRepo, options) {
|
|
|
172
172
|
console.log(` Graph ID: ${graphId}`);
|
|
173
173
|
console.log(` Commit: ${gitInfo.commitHash} (${gitInfo.branch})`);
|
|
174
174
|
console.log(` Message: ${gitInfo.message}`);
|
|
175
|
-
//
|
|
175
|
+
// Step 1: Fingerprint
|
|
176
176
|
console.log("\n[1/5] Fingerprinting project...");
|
|
177
177
|
const fingerprint = analyzeFingerprint(absoluteRepoPath);
|
|
178
178
|
console.log(` Framework: ${fingerprint.framework} | Language: ${fingerprint.language} | Type: ${fingerprint.projectType}`);
|
|
179
|
-
//
|
|
179
|
+
// Step 2: Filesystem / routes
|
|
180
180
|
console.log("\n[2/5] Analyzing filesystem routes...");
|
|
181
181
|
const routes = analyzeFilesystem(absoluteRepoPath, fingerprint);
|
|
182
182
|
console.log(` Routes found: ${routes.length}`);
|
|
@@ -184,21 +184,20 @@ export async function analyzePipeline(repoPath, isGithubRepo, options) {
|
|
|
184
184
|
// It is important to add here before the detection of the edges
|
|
185
185
|
let routeNodes = routesToCodeNodes(routes, absoluteRepoPath);
|
|
186
186
|
console.log(` Route nodes created: ${routeNodes.length}`);
|
|
187
|
-
//
|
|
187
|
+
// Step 3: Parse source files into nodes
|
|
188
188
|
console.log("\n[3/5] Parsing source files...");
|
|
189
189
|
const parserResult = parseRepo(absoluteRepoPath);
|
|
190
190
|
console.log(` Files: ${parserResult.stats.totalFiles} | Nodes: ${parserResult.stats.totalNodes} | Skipped: ${parserResult.stats.skippedFiles}`);
|
|
191
|
-
//
|
|
191
|
+
// Step 3.5: Build third-party nodes
|
|
192
192
|
const thirdPartyNodes = options?.includedThirdPartyLibs?.length
|
|
193
193
|
? buildThirdPartyNodes(absoluteRepoPath, options.includedThirdPartyLibs)
|
|
194
194
|
: [];
|
|
195
195
|
if (thirdPartyNodes.length) {
|
|
196
196
|
console.log(` Third-party nodes: ${thirdPartyNodes.length}`);
|
|
197
197
|
}
|
|
198
|
-
//
|
|
198
|
+
// Step 4: Detect edges
|
|
199
199
|
console.log("\n[4/5] Detecting edges...");
|
|
200
200
|
const edgeResult = detectEdges([...parserResult.nodes, ...routeNodes, ...thirdPartyNodes], routes, absoluteRepoPath, fingerprint);
|
|
201
|
-
// filter API_ROUTE nodes without handlers - because at the time of converting routes to code nodes, POST and GET both possibilties are taken for the API_ROUTE nodes, however it is possible that only one of them is being used for that route. Meaning only one handler and for the second method undefined handler.
|
|
202
201
|
routeNodes = routeNodes.filter(routeNode => {
|
|
203
202
|
if (routeNode.metadata.routeNodeType === "API_ROUTE") {
|
|
204
203
|
const hasHandler = edgeResult.edges.some(edge => edge.type === "HANDLES" && edge.from === routeNode.id);
|
|
@@ -208,7 +207,7 @@ export async function analyzePipeline(repoPath, isGithubRepo, options) {
|
|
|
208
207
|
});
|
|
209
208
|
const allNodes = [...parserResult.nodes, ...routeNodes, ...thirdPartyNodes, ...edgeResult.ghostNodes];
|
|
210
209
|
const allEdges = edgeResult.edges;
|
|
211
|
-
//
|
|
210
|
+
// Step 5: Score and filter
|
|
212
211
|
console.log("\n[5/5] Scoring and filtering...");
|
|
213
212
|
const scoringResult = scoreAndFilter(allNodes, allEdges, options?.thresholds);
|
|
214
213
|
const nodeScores = mapToRecord(scoringResult.nodeScores);
|
|
@@ -236,7 +235,7 @@ export async function analyzePipeline(repoPath, isGithubRepo, options) {
|
|
|
236
235
|
gitInfo,
|
|
237
236
|
};
|
|
238
237
|
}
|
|
239
|
-
//
|
|
238
|
+
// refilterPipeline
|
|
240
239
|
export function refilterPipeline(stored, thresholds) {
|
|
241
240
|
const existingScores = new Map(Object.entries(stored.nodeScores));
|
|
242
241
|
const scoringResult = scoreAndFilter(stored.allNodes, stored.allEdges, thresholds, existingScores);
|
package/dist/types.d.ts
CHANGED
|
@@ -69,7 +69,7 @@ export interface CodeNode {
|
|
|
69
69
|
score?: Number;
|
|
70
70
|
metadata: Record<string, unknown>;
|
|
71
71
|
}
|
|
72
|
-
export type EdgeType = "CALLS" | "IMPORTS" | "READS_FROM" | "WRITES_TO" | "PROP_PASS" | "EMITS" | "LISTENS" | "WRAPPED_BY" | "GUARDS" | "HANDLES" | "TESTS" | "USES";
|
|
72
|
+
export type EdgeType = "CALLS" | "IMPORTS" | "READS_FROM" | "WRITES_TO" | "PROP_PASS" | "EMITS" | "LISTENS" | "WRAPPED_BY" | "GUARDS" | "HANDLES" | "TESTS" | "USES" | "NEXTJS_API_CALL";
|
|
73
73
|
export interface CodeEdge {
|
|
74
74
|
from: string;
|
|
75
75
|
to: string;
|