backlens 0.0.0-stage → 0.1.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shayan Dutta
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,90 @@
-# Temporary Holding Version
+# backlens
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+See every request your Node backend handles, every query it makes, and what is wrong with them. No code changes, no account, nothing leaves your machine.
+
+```
+npx backlens
+```
+
+Run it in your server's folder. Your browser opens the dashboard; the first time, pick which package.json script starts the app and which runs the tests. From then on everything happens in the browser: start, stop and restart the app, read its logs, run the tests, accept known issues, compare before and after.
+
+The link backlens prints carries a private token: keep it to yourself, it controls your app. The dashboard listens on 127.0.0.1 only, refuses requests from other sites, and only ever runs scripts from your package.json by name.
+
+To run one exact command instead: `npx backlens -- node server.js`.
+
+To keep it in the project, install it as a dev dependency with your package manager (`npm i -D backlens`, `pnpm add -D backlens`, `yarn add -D backlens`). In a pnpm or yarn workspace, use that tool: npm cannot install into a project whose package.json uses `workspace:` versions.
+
+The dashboard uses port 7777, or the next free one if it is taken. `--port` picks one.
+
+## What you get
+
+- **Routes**: every endpoint your app registers (fastify and nest report them at startup; express routes appear once called), with calls, p50 and p95, and how many database, Redis and outgoing HTTP calls each request makes.
+- **Requests**: a waterfall of any single request, with the line of your code behind each call.
+- **Issues**: problems found in real traffic, worst first, each pointing at the code that caused it:
+  - **Repeated calls**: the same query with the same values twice in one request. backlens names the two places the code paths part, which is where the second fetch comes from.
+  - **N+1**: one query run three or more times with different values in one request.
+  - **Calls that wait for each other**: two or more calls of the same kind, awaited back to back in one function. Shown as a hint: if they don't depend on each other, `Promise.all` runs them together.
+
+## In CI: fail the build on new issues
+
+```
+npx backlens test -- npx vitest run
+```
+
+Runs your test command with recording on, prints every issue it found, and exits 1 when there is a new warning. Tests that call the app over HTTP (supertest, fetch) are recorded like real traffic.
+
+- `--update-baseline` accepts everything found in this run as known, in `backlens.baseline.json`. Commit that file. An issue keeps its identity when lines above it move.
+- `--strict` fails on new hints too.
+- `--json report.json` writes the report for a CI artifact.
+
+```
+backlens  90 requests (207 queries, 241 redis, 0 http out), 11 of 47 routes called, 3 issues, 2 new
+  new   3 calls are made twice in one request  GET /v1/lessons/:id, 5/11 requests
+        src/modules/access/access.service.ts:75 → src/modules/access/access.service.ts:55
+  36 routes never called by these tests
+backlens: 2 new issues. Fix them, or accept with --update-baseline.
+```
+
+## Missing indexes
+
+For each distinct SELECT, backlens asks Postgres for its plan with `EXPLAIN (GENERIC_PLAN)` (Postgres 16+). Plain EXPLAIN never runs the query, and the generic plan needs no parameter values. When a table is read row by row while filtering on columns that no index starts with, the Issues tab names the table and columns and gives the `CREATE INDEX` to add. It checks whether an index exists, not how fast the query was, so it is right on a small dev database too.
+
+## Before and after
+
+In the Compare tab, save a snapshot, change your code, restart, use the same pages, and compare per route: queries and Redis calls per request, p50 and issues. Snapshots are kept in `.backlens/snapshots` in your project; add `.backlens/` to `.gitignore`. Timings only count as a change with 5 or more calls on both sides.
+
+## Explain and fix, with proof
+
+Every issue has **Explain** and **Suggest fix**. A fix goes: a preview of exactly what would be sent and to whom, the model's answer as a diff, **Apply and check** (only on files with no uncommitted changes, with a backup), then backlens rebuilds if needed, restarts your app, replays the request that showed the issue and runs your tests. Only a fix that removes the issue and keeps the tests green is marked proven. Keep it, or revert to the exact files you had.
+
+Who answers is chosen per issue, with the reasons shown:
+
+- **Local model** (free, private): any model in [Ollama](https://ollama.com). Picked by default when Ollama is running.
+- **Recorded answers** (free): no model; fixes come from `.backlens/ai-fixtures.json`. For trying the flow and for tests.
+- **Claude with your own key**: set `ANTHROPIC_API_KEY` before starting backlens. Explaining uses Claude Haiku 5.5; fixes use Sonnet 5.5 or Opus 5.5 depending on how hard the issue looks (files involved, issue type, test coverage, your past results), one step up after a failed check. Missing indexes never use a model. A daily budget (default $1) is enforced before sending.
+
+Values that look like secrets are masked, `.env` files and `node_modules` are never read, and requests kept for replay stay in memory only. Replaying a POST, PUT or DELETE runs it again against your dev database, so it is off unless you allow it in AI settings.
+
+## How it works
+
+`backlens` starts the dashboard, then runs your command with `NODE_OPTIONS=--require backlens/register`. Inside your app, the register script:
+
+- times each incoming request on Node's own HTTP server, so it works with express, fastify, nest, Next.js and plain `http`, and records requests the client cancels. Framework internals such as `/_next/` assets and favicons are skipped;
+- wraps `pg` (Client and Pool), `ioredis` and global `fetch`;
+- records where each call came from. For ORMs like drizzle, which call the driver from a microtask, the call site is read from V8's async stack once the query settles. Code run from bundled chunks (Next.js with Turbopack or webpack) is traced back to your source files through their source maps.
+
+Query values are never sent, only a short hash so identical questions can be told apart from different ones. Emails, IP addresses and ids in Redis keys are masked. The dashboard listens on 127.0.0.1 only and keeps the last 1000 requests in memory.
+
+## Limits of this version
+
+- Tested on macOS with Node 20+, on a nest + fastify + drizzle app and a Next.js 16 + drizzle app. Linux should work the same way; Windows is untested.
+- Requests, `pg` and `ioredis` are recorded in CommonJS and ESM apps (vitest included). The up-front route list needs fastify loaded through CommonJS, as nest does.
+- `pg`, `ioredis` and `fetch` only. No Prisma, mysql or mongo yet.
+- The index advisor needs Postgres 16 or newer, and looks at equality filters only.
+- The "wait for each other" hint skips calls whose input matches an earlier call's output, but it cannot see every dependency (a value changed before use, an order that matters for side effects), so read it as a question, not a verdict.
+
+## Develop
+
+```
+node --test test/
+```

package/ai/context.js

@@ -0,0 +1,151 @@
+import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
+import { join, relative, resolve, sep } from 'node:path';
+import { tablesOf } from '../analyze.js';
+
+/**
+ * Collects only the code an issue is about: the whole function around each place backlens
+ * recorded, plus how the tables involved are defined. Never reads .env files, node_modules or
+ * anything outside the project, and masks secrets in what it does read.
+ */
+
+const MAX_FILE = 1024 * 1024;
+const MAX_LINES = 700;
+const BLOCKED = /(^|[/\\])(node_modules|\.git|\.backlens|dist|build|coverage)([/\\]|$)|(^|[/\\])\.env/;
+const FUNCTION_START = /^\s*(export\s+)?(default\s+)?(async\s+)?(function\b|(public|private|protected|static|async|get|set)\s+[\w$]+\s*\(|(?!(?:if|for|while|switch|catch|with|return|await|else|do|try|new|typeof)\b)[\w$]+\s*\([^)]*\)\s*(:[^={]+)?\{|(const|let|var)\s+[\w$]+\s*=\s*(async\s*)?(\([^)]*\)|[\w$]+)\s*=>|[\w$]+\s*:\s*(async\s*)?(function|\())/;
+
+export function insideProject(root, file) {
+  const abs = resolve(root, file);
+  return (abs === root || abs.startsWith(root + sep)) && !BLOCKED.test(relative(root, abs)) ? abs : null;
+}
+
+/** Masks values that look like secrets, so they never reach a model. */
+export function scrub(text) {
+  return text
+    .replace(/((?:api[_-]?key|secret|token|password|passwd|private[_-]?key|auth)\w*["']?\s*[:=]\s*)(["'`])[^"'`\n]{6,}\2/gi, '$1$2***$2')
+    .replace(/\b([a-z][a-z0-9+.-]*:\/\/[^:/\s"'`]+:)[^@\s"'`]+@/gi, '$1***@')
+    .replace(/\b(sk|pk|rk)[-_](live|test|ant|proj)?[-_]?[A-Za-z0-9_-]{16,}/g, '***')
+    .replace(/\bAKIA[0-9A-Z]{16}\b/g, '***')
+    .replace(/-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----/g, '***');
+}
+
+const indent = (line) => line.match(/^\s*/)[0].length;
+
+/** The enclosing function of a line: back to its signature, forward to its closing brace. */
+export function enclosing(lines, lineNo) {
+  const at = Math.min(Math.max(lineNo - 1, 0), lines.length - 1);
+  let start = Math.max(0, at - 30);
+  for (let i = at; i >= Math.max(0, at - 120); i--) {
+    if (FUNCTION_START.test(lines[i]) && indent(lines[i]) <= indent(lines[at])) {
+      start = i;
+      break;
+    }
+  }
+  let depth = 0;
+  let seen = false;
+  let end = Math.min(lines.length - 1, at + 30);
+  for (let i = start; i < Math.min(lines.length, start + 160); i++) {
+    for (const ch of lines[i].replace(/(["'`])(?:\\.|(?!\1).)*\1/g, '').replace(/\/\/.*$/, '')) {
+      if (ch === '{') {
+        depth++;
+        seen = true;
+      } else if (ch === '}') depth--;
+    }
+    if (seen && depth <= 0) {
+      end = i;
+      break;
+    }
+  }
+  return { from: start + 1, to: Math.max(end + 1, lineNo) };
+}
+
+function walk(dir, root, out, limit = 3000) {
+  let entries;
+  try {
+    entries = readdirSync(dir, { withFileTypes: true });
+  } catch {
+    return;
+  }
+  for (const e of entries) {
+    if (out.length >= limit) return;
+    const p = join(dir, e.name);
+    if (BLOCKED.test(relative(root, p))) continue;
+    if (e.isDirectory()) walk(p, root, out, limit);
+    else if (/\.(ts|js|mjs|cjs|prisma)$/.test(e.name)) out.push(p);
+  }
+}
+
+/** Where each table is defined (drizzle pgTable/mysqlTable, prisma model), to show its columns and indexes. */
+function schemaFor(root, tables) {
+  if (!tables.length) return [];
+  const files = [];
+  walk(root, root, files);
+  const found = [];
+  for (const file of files) {
+    let text;
+    try {
+      if (statSync(file).size > MAX_FILE) continue;
+      text = readFileSync(file, 'utf8');
+    } catch {
+      continue;
+    }
+    const lines = text.split('\n');
+    for (const t of tables) {
+      const re = new RegExp(`(Table\\(\\s*["'\`]${t}["'\`]|@@map\\(\\s*"${t}"\\)|^model\\s+${t}\\b)`, 'i');
+      const i = lines.findIndex((l) => re.test(l));
+      if (i === -1 || found.some((f) => f.table === t)) continue;
+      const { from, to } = enclosing(lines, i + 1);
+      found.push({ table: t, path: relative(root, file), from: Math.min(from, i + 1), to: Math.min(to, i + 40) });
+    }
+  }
+  return found;
+}
+
+/** The context for one issue, with a preview of exactly what would be sent. */
+export function buildContext(root, issue) {
+  const wanted = new Map();
+  const add = (file, from, to) => {
+    const list = wanted.get(file) || [];
+    list.push({ from, to });
+    wanted.set(file, list);
+  };
+  const frameSets = (issue.frames || []).filter(Boolean);
+  const sites = new Set();
+  for (const frames of frameSets) for (const f of frames.slice(0, 3)) sites.add(`${f.file}:${f.line}`);
+  for (const s of issue.sites || []) sites.add(s);
+
+  for (const site of sites) {
+    const m = site.match(/^(.*):(\d+)$/);
+    if (!m) continue;
+    const abs = insideProject(root, m[1]);
+    if (!abs || !existsSync(abs) || statSync(abs).size > MAX_FILE) continue;
+    const lines = readFileSync(abs, 'utf8').split('\n');
+    const { from, to } = enclosing(lines, Number(m[2]));
+    add(relative(root, abs), from, to);
+  }
+  const tables = new Set();
+  for (const line of String(issue.sql || '').split('\n')) for (const t of tablesOf(line).tables) tables.add(t);
+  for (const s of schemaFor(root, [...tables])) add(s.path, s.from, s.to);
+
+  // Merge overlapping ranges per file, then cut to the overall budget.
+  const files = [];
+  let budget = MAX_LINES;
+  for (const [path, ranges] of wanted) {
+    ranges.sort((a, b) => a.from - b.from);
+    const merged = [];
+    for (const r of ranges) {
+      const last = merged.at(-1);
+      if (last && r.from <= last.to + 3) last.to = Math.max(last.to, r.to);
+      else merged.push({ ...r });
+    }
+    const lines = readFileSync(join(root, path), 'utf8').split('\n');
+    for (const r of merged) {
+      if (budget <= 0) break;
+      const to = Math.min(r.to, r.from + budget - 1);
+      const text = lines.slice(r.from - 1, to).map((l, i) => `${String(r.from + i).padStart(5)}  ${l}`).join('\n');
+      files.push({ path, from: r.from, to, text: scrub(text) });
+      budget -= to - r.from + 1;
+    }
+  }
+  const chars = files.reduce((n, f) => n + f.text.length, 0) + String(issue.sql || '').length + 2000;
+  return { files, tokens: Math.round(chars / 3.5), preview: files.map((f) => ({ path: f.path, from: f.from, to: f.to })) };
+}