seer-mcp 0.1.0 → 0.1.2

package/adding-new-languages.md

@@ -0,0 +1,647 @@
+# Adding New Languages To Seer
+
+This is the full engineering checklist for adding a programming language to
+Seer-Core. Seer language support is not just "parse a file and grab function
+names"; a language extension touches discovery, Tree-sitter grammar loading,
+symbol extraction, call attribution, imports, optional routes/config/service
+calls, complexity, cache behavior, and tests.
+
+The current language surface is:
+
+- Python
+- JavaScript
+- TypeScript / TSX / JSX
+- Go
+- Java
+- Rust
+- C
+- C++
+- C#
+
+Core files to read before starting:
+
+- `src/types.ts`
+- `src/parser/walker.ts`
+- `src/parser/parserContext.ts`
+- `src/parser/index.ts`
+- `src/parser/languages/*.ts`
+- `src/indexer/index.ts`
+- `src/db/store.ts`
+
+## 1. Decide The Scope
+
+Before writing code, decide what "language support" means for this language.
+At minimum, Seer needs:
+
+- file extension detection
+- Tree-sitter grammar loading
+- definitions
+- call/reference names
+- imports
+- basic complexity metrics
+- fixture coverage
+- smoke and parser parity tests
+
+Optional but important for agent usefulness:
+
+- routes/endpoints
+- config/env reads
+- outbound service calls
+- declarations vs definitions
+- test-file conventions
+- package/dependency manifest support
+- scale-test coverage on a real repository
+
+Do not start by adding every possible AST node. Seer defaults should optimize
+for AI navigation: first-party, non-generated, rankable definitions and useful
+edges. Type aliases, declarations, and type-reference rows are useful only when
+they are role-labelled and filtered correctly.
+
+## 2. Confirm The Tree-Sitter Grammar Exists
+
+Seer loads WASM grammars from `tree-sitter-wasms`:
+
+```text
+tree-sitter-wasms/out/tree-sitter-<grammar>.wasm
+```
+
+Check the installed package before wiring a language:
+
+```powershell
+Get-ChildItem node_modules\tree-sitter-wasms\out | Select-String "<language>"
+```
+
+If the grammar does not exist, do not just add an extension mapping. You either
+need to add a dependency that ships the WASM grammar or defer the language.
+
+Some languages need grammar aliases:
+
+- TypeScript uses `typescript`.
+- TSX uses `tsx`.
+- JavaScript/JSX use `javascript`.
+- C# uses `c_sharp`.
+- C and C++ use different grammars but share one C-family extractor.
+
+## 3. Update Shared Types
+
+Edit `src/types.ts`:
+
+1. Add the language to the `Language` union.
+2. If the language needs new symbol kinds, update `SymbolKind`. Prefer existing
+   kinds first: `function`, `method`, `constructor`, `class`, `struct`, `enum`,
+   `interface`, `type`, `variable`.
+3. If the language introduces a genuinely new edge kind, update `EdgeKind`,
+   schema, Store methods, and tests. Avoid this unless necessary.
+4. If it adds a new service protocol, update `ServiceProtocol`, route/service
+   call fields, schema persistence, MCP/CLI output, and Track G/H tests.
+
+Most new languages should only need a new `Language` value.
+
+## 4. Add The Extractor
+
+Create:
+
+```text
+src/parser/languages/<language>.ts
+```
+
+Export a `LanguageExtractor`:
+
+```ts
+import type Parser from 'web-tree-sitter';
+import type { SymbolDef } from '../../types.js';
+import type { LanguageExtractor } from '../walker.js';
+import { firstLine } from '../walker.js';
+
+const BRANCH_NODES = new Set<string>([
+  'if_statement',
+  'while_statement',
+]);
+
+const NESTING_NODES = new Set<string>([
+  'if_statement',
+  'while_statement',
+]);
+
+const CANDIDATE_NODE_TYPES = [
+  'function_definition',
+  'call_expression',
+  'import_statement',
+] as const;
+
+export const myLanguageExtractor: LanguageExtractor = {
+  languageName: 'my_language',
+  extensions: ['.mine'],
+  branchNodeTypes: BRANCH_NODES,
+  nestingNodeTypes: NESTING_NODES,
+  candidateNodeTypes: CANDIDATE_NODE_TYPES,
+
+  tryExtractDefinition(node: Parser.SyntaxNode): SymbolDef | null {
+    return null;
+  },
+
+  tryExtractCallName(node: Parser.SyntaxNode): string | null {
+    return null;
+  },
+
+  tryExtractImport(node: Parser.SyntaxNode): string | null {
+    return null;
+  },
+};
+```
+
+### Extractor Contract
+
+`LanguageExtractor` is defined in `src/parser/walker.ts`.
+
+Implement these required methods:
+
+- `tryExtractDefinition(node)`
+- `tryExtractCallName(node)`
+- `tryExtractImport(node)`
+
+Optional methods:
+
+- `tryExtractContextName(node)`
+- `tryExtractRoute(node)`
+- `tryExtractConfigKey(node)`
+- `tryExtractServiceCalls(node)`
+
+Optional complexity fields:
+
+- `branchNodeTypes`
+- `nestingNodeTypes`
+- `candidateNodeTypes`
+
+The extractor should return `null` unless it is certain. Conservative misses
+are better than noisy symbols that pollute PageRank, search, and module
+clustering.
+
+## 5. Wire ParserContext
+
+Edit `src/parser/parserContext.ts`:
+
+1. Import the extractor.
+2. Add extensions to `EXT_TO_LANGUAGE`.
+3. Add the extractor to `EXTRACTORS`.
+4. Add the grammar name to `GRAMMAR_NAME`.
+5. Update `grammarForExtension()` if the language has multiple grammar variants.
+
+Example:
+
+```ts
+import { myLanguageExtractor } from './languages/my_language.js';
+
+export const EXT_TO_LANGUAGE = {
+  '.mine': 'my_language',
+};
+
+export const EXTRACTORS = {
+  my_language: myLanguageExtractor,
+};
+
+export const GRAMMAR_NAME = {
+  my_language: 'my_language',
+};
+```
+
+If one extractor supports multiple grammars, follow the TypeScript/JavaScript
+or C/C++ patterns. Do not duplicate logic just because extensions differ.
+
+## 6. Definitions: The Hard Part
+
+Definitions must be stable, useful, and not too noisy.
+
+Rules:
+
+- Return short names only. The walker computes `qualifiedName`.
+- Use `symbolRole='definition'` for body-bearing canonical definitions.
+- Use `symbolRole='declaration'` for prototypes/forward declarations.
+- Use `symbolRole='type_ref'` only if you intentionally index bare type
+  references and also add query filters/tests for them.
+- Do not index local variables as symbols unless there is a strong agent use
+  case.
+- Prefer body-bearing definitions for rankable rows.
+
+Rankable symbols are generally:
+
+- `function`
+- `method`
+- `constructor`
+- `class`
+
+Non-rankable or lower-value symbols should still be queryable only when useful,
+but should not dominate PageRank/search defaults.
+
+### C/C++ Pitfall: Body Gates
+
+The C/C++ extractor had a real large-repo bug:
+
+```c
+int foo(struct device *dev);
+```
+
+Tree-sitter parses `struct device` as a `struct_specifier`, the same node type
+used by a real definition:
+
+```c
+struct device {
+  int id;
+};
+```
+
+Without a body check, Seer emitted tens of thousands of fake `struct device`
+symbols on Linux. The fix in `src/parser/languages/cpp.ts` is:
+
+```ts
+if (!node.childForFieldName('body')) return null;
+```
+
+Apply the same discipline to any language where the grammar reuses definition
+node types for references, declarations, or type annotations.
+
+Examples to watch for:
+
+- C/C++ `struct_specifier`, `class_specifier`, `enum_specifier`, `union_specifier`
+- languages where `type_identifier` appears both in declarations and references
+- annotations/decorators that look like calls but are not behavior
+- interface declarations that contain method signatures but no method bodies
+- generated partial classes or metadata-only declarations
+
+### Declarations vs Definitions
+
+If a language has declaration sites separate from implementation sites, label
+them:
+
+```ts
+const def = mkDef(name, 'function', node);
+def.symbolRole = 'declaration';
+return def;
+```
+
+Then add tests that default queries hide the declaration and
+`includeDeclarations=true` brings it back.
+
+## 7. Qualified Names And Overloads
+
+Do not manually build `qualifiedName` in the extractor. The walker owns it.
+
+The walker keeps a definition stack and appends `#N` when sibling definitions
+share the same short name:
+
+```text
+Overload.run
+Overload.run#1
+```
+
+This protects Java/C++/C#/TypeScript overloads and same-short-name methods like
+`Alpha.run` and `Beta.run`.
+
+Use `tryExtractContextName()` only for scopes that provide naming context but
+are not themselves symbols. Rust `impl AuthService { ... }` is the model:
+
+- the `impl` block is not a symbol
+- it should still push `AuthService` so methods become `AuthService.login`
+
+Context names should usually be raw names. They should not increment the
+sibling overload counters unless they are actual definitions.
+
+## 8. Calls And Caller Attribution
+
+`tryExtractCallName(node)` should return the callee short name only:
+
+```ts
+foo()           -> "foo"
+obj.method()    -> "method"
+new User()      -> "User"
+Namespace.run() -> "run"
+```
+
+The walker sets `callerName` from the current qualified definition stack. This
+is important: older bugs came from using only the short caller name, which made
+`Alpha.run` and `Beta.run` collapse into one caller.
+
+Do not attempt type inference inside the extractor. Seer resolves what it can
+after all files are parsed. If a call cannot be resolved, keeping the unresolved
+edge is still useful.
+
+## 9. Imports
+
+`tryExtractImport(node)` should return the module/path exactly enough for the
+indexer to resolve or display it:
+
+- relative imports: `./foo`, `../bar`
+- package imports: `react`, `lodash`
+- language module imports: `java.util.List`, `github.com/x/y`
+
+If the language has non-standard import resolution, add or update resolver
+tests. Do not assume Java/Go/Rust package systems behave like TS/Python
+relative imports.
+
+## 10. Routes, Config, And Service Calls
+
+These are optional but valuable.
+
+### Routes
+
+`tryExtractRoute(node)` returns `RouteDef[]`.
+
+Use it for server-side registrations:
+
+```text
+app.get('/users', handler)
+@router.post('/users')
+@GetMapping("/users")
+```
+
+Rules:
+
+- only emit deterministic routes
+- record framework
+- record handler name when statically known
+- handle class/router prefixes carefully
+- do not emit class-level route prefixes as standalone routes
+
+The Spring class-level `@RequestMapping("/api")` bug is the cautionary tale:
+class annotations should prefix method routes, not create bogus independent
+routes.
+
+### Config Keys
+
+`tryExtractConfigKey(node)` returns `ConfigKeyRead`.
+
+Examples:
+
+```text
+process.env.DB_URL
+os.getenv("DB_URL")
+System.getenv("DB_URL")
+```
+
+The walker fills `callerName` from the def stack.
+
+### Service Calls
+
+`tryExtractServiceCalls(node)` returns `ServiceCallDef[]`.
+
+Use it for outbound client calls:
+
+```text
+fetch('/api/users')
+axios.post('/checkout')
+requests.get('/health')
+http.Get("http://payment/api/charge")
+```
+
+Rules:
+
+- server-side route registrations are not service calls
+- keep `rawTarget` even when the target is dynamic
+- fill `normalizedPath` only when statically extractable
+- record method/protocol/host hints when known
+- stay conservative; false links are worse than unresolved calls
+
+The walker skips service-call extraction on nodes already classified as route
+registrations, so a route mount does not link to itself.
+
+## 11. Candidate Node Types
+
+Every extractor should declare `candidateNodeTypes`.
+
+This list must be a superset of every node type any extractor method may
+accept:
+
+- definitions
+- calls
+- imports
+- context nodes
+- routes
+- config reads
+- service calls
+
+Missing a node type is dangerous because the Tree-sitter query path will never
+call the extractor for that node. The baseline walker may still work, which is
+why `tests/query-parity.ts` exists.
+
+For grammars shared by multiple languages, a superset is okay. C/C++ uses one
+candidate list even though some node types only exist in the C++ grammar; the
+parser probes node types and drops those rejected by the active grammar.
+
+When you add or change candidate nodes, run:
+
+```powershell
+npm run test:query-parity
+```
+
+Also consider extending `tests/query-parity.ts` if the new feature category is
+not included in its canonical comparison.
+
+## 12. Complexity Metrics
+
+If the language has function-like bodies, add `branchNodeTypes` and
+`nestingNodeTypes`.
+
+Cyclomatic complexity increments on branches like:
+
+- `if`
+- `for`
+- `while`
+- `catch`
+- `case`
+- ternary/conditional
+
+Cognitive complexity uses nesting depth. Keep these lists conservative and
+language-specific.
+
+If `branchNodeTypes` is omitted, Seer still records LOC for function-like
+symbols, but leaves cyclomatic/cognitive/max nesting null.
+
+## 13. Cache And Migration Gotchas
+
+Changing an extractor does not automatically update already-cached files. If
+file hashes are unchanged, `upsertFileWithCache()` can skip parsing.
+
+This matters when a new language or new extraction category is introduced after
+DBs already exist. Track G hit this exact bug: migrated pre-v8 DBs had empty
+`service_calls` because unchanged files were hash-cached before service-call
+extraction existed.
+
+If a new extractor feature needs to backfill existing DBs, add a marker:
+
+- store a version or completion marker in schema metadata/state
+- detect missing backfill before indexing
+- bypass parse cache for one full pass
+- mark backfill complete only after the post-pass succeeds
+
+Patterns to inspect:
+
+- shape-hash backfill in Track F
+- service-call backfill in Track G
+- `Store.hasMissingShapeHashes()`
+- `Store.needsServiceCallBackfill()`
+
+## 14. Discovery And Classification
+
+Most languages only need extension mapping. But check whether the language has:
+
+- generated files with conventional suffixes
+- vendored package roots
+- build-output roots
+- test-file naming conventions
+
+Update these if necessary:
+
+- `src/indexer/discovery.ts`
+- `src/indexer/classify.ts`
+- scale-test role assertions if the language appears in large repos
+
+Default queries should stay project-first and should not let vendor/generated
+symbols dominate PageRank.
+
+## 15. Tests To Add
+
+At minimum, add one fixture file:
+
+```text
+tests/fixtures/sample.<ext>
+```
+
+It should include:
+
+- at least one top-level function
+- at least one class/struct/module-level container if the language has one
+- at least one method
+- at least one call that resolves
+- at least one call that remains unresolved
+- at least one import
+- one same-short-name collision or overload case if the language supports it
+- one declaration-vs-definition case if the language has declarations
+
+Then update or add tests:
+
+### Smoke
+
+Update `tests/smoke.ts`:
+
+- assert the language appears in `store.getStats().languages`
+- assert important symbols are found
+- assert at least one caller/callee relationship
+- assert qualified names are correct
+- assert overloads/collisions do not collapse
+
+Run:
+
+```powershell
+npm run test:smoke
+```
+
+### Query Parity
+
+Update `tests/query-parity.ts` extension allowlist and fixture coverage.
+
+Run:
+
+```powershell
+npm run test:query-parity
+```
+
+### Worker Parity
+
+Parser workers must produce byte-identical extraction to in-process parsing.
+If the fixture walker does not already pick up the new extension, update it.
+
+Run:
+
+```powershell
+npm run test:worker-parity
+```
+
+### Filters
+
+If the language emits declarations, type refs, tests, vendor/generated files,
+or non-rankable symbol kinds, update:
+
+```powershell
+npm run test:filters
+```
+
+### Feature Tests
+
+If the language adds routes/config/service calls, add focused tests in the
+relevant suite:
+
+- `tests/trackcd.ts` for routes/config/deps/complexity
+- `tests/trackg.ts` for service calls/links
+- `tests/mcp-trackg.ts` if the feature is exposed through MCP
+
+### Full Suite
+
+Always end with:
+
+```powershell
+npm run build
+npm test
+```
+
+If the new language is intended for large repos, also run:
+
+```powershell
+npm run test:scale-parallel-parity
+npm run scale-test -- --only <repo>
+```
+
+## 16. Manual Inspection Checklist
+
+After indexing a fixture or real repo, inspect:
+
+- total files by language
+- top symbols by PageRank
+- role counts
+- declarations hidden by default
+- test symbols hidden by default
+- vendor/generated paths not appearing in top results
+- qualified names for methods
+- callers/callees on same-short-name methods
+- service calls not duplicating server route registrations
+- route prefixes applied correctly
+- cache re-run does not delete or duplicate rows
+
+Useful commands:
+
+```powershell
+npm run dev -- index tests/fixtures --reset
+npm run dev -- stats
+npm run dev -- symbols <name> --include-declarations
+npm run dev -- callers <name> --limit 20
+npm run dev -- callees <name> --limit 20
+npm run dev -- routes
+npm run dev -- service-calls
+npm run dev -- service-links
+```
+
+## 17. Final Review Checklist
+
+Before calling the language done:
+
+- `Language` union updated
+- extension detection works
+- grammar loads from `tree-sitter-wasms`
+- extractor added and wired in `EXTRACTORS`
+- `GRAMMAR_NAME` and `grammarForExtension()` are correct
+- `candidateNodeTypes` is a true superset
+- body-bearing definitions are not confused with references/declarations
+- declarations use `symbolRole='declaration'`
+- noisy type refs are avoided or role-labelled
+- qualified names come from the walker, not extractor-local strings
+- overloads/same-short-name symbols stay distinct
+- calls get correct caller attribution
+- imports are captured
+- complexity is populated for function-like symbols
+- optional route/config/service extraction is conservative and tested
+- cache/backfill implications are handled
+- smoke/query-parity/worker-parity/full suite pass
+- scale/parity tests pass if the language targets large repos
+
+The guiding principle: add enough language intelligence for agents to navigate
+with confidence, while avoiding noisy rows that make the graph look bigger but
+less true.

package/dist/cli/index.js

@@ -46,7 +46,7 @@ const behavior_js_1 = require("../indexer/behavior.js");
 const risk_js_1 = require("../indexer/risk.js");
 const context_js_1 = require("../indexer/context.js");
 const init_js_1 = require("./init.js");
-const VERSION = '0.1.0';
+const VERSION = '0.1.2';
 const KNOWN_CLIENTS = ['claude', 'cursor', 'vscode', 'codex', 'gemini', 'antigravity'];
 function resolveDb(repoPath, customDb) {
     if (customDb)

package/package.json

@@ -1,7 +1,8 @@
 {
   "name": "seer-mcp",
-  "version": "0.1.0",
-  "description": "Local-first AI codebase explainer — onboarding from months to days",
+  "version": "0.1.2",
+  "description": "Give your AI agents a map of your repo before they edit.",
+  "license": "Apache-2.0",
   "private": false,
   "type": "commonjs",
   "bin": {