claude-db 0.8.3 → 0.8.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,76 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.8.5
4
+
5
+ ### Added
6
+
7
+ - **Using a type now produces a graph edge.**
8
+ ([#19](https://github.com/Avijit07x/claude-db/issues/19))
9
+ A type in argument, return, annotation or type-argument position matched no
10
+ rule, so all 985 interfaces and type aliases on one 214k-line repo had zero
11
+ inbound edges and `find_usages` reported every one of them as unused.
12
+ `type_identifier` is now a reference rule. Matching it directly would also
13
+ hit declaration sites and emit a self-edge for each, so `ReferenceRule` takes
14
+ an optional `excludeParents`, and the rule skips the seven parents that mean
15
+ a type is being declared rather than used. Measured on this repo: 2,387
16
+ reference edges and zero self-edges. This completes the second half of the
17
+ issue; rendering a component shipped in 0.8.4.
18
+
19
+ ### Fixed
20
+
21
+ - **Two declarations sharing a name and kind in one file collapsed into one
22
+ symbol.** ([#18](https://github.com/Avijit07x/claude-db/issues/18))
23
+ `symbolId` hashed project, file, name and kind with nothing positional, so
24
+ any two functions in a file that each declared `const total` produced a
25
+ single symbol and one of them was lost. On claude-db's own source that was 89
26
+ colliding groups and 115 symbols never stored. The id now carries the
27
+ occurrence index of the name within its file. The first occurrence keeps the
28
+ id it always had, so existing graphs do not churn; only the second and later
29
+ duplicates gain new ids. `SCAN_VERSION` is bumped so stored graphs rebuild.
30
+
31
+ ## 0.8.4
32
+
33
+ ### Fixed
34
+
35
+ - **Postgres: `scan` issued one round-trip per symbol and per edge, making it
36
+ unusable against a hosted database.**
37
+ ([#16](https://github.com/Avijit07x/claude-db/issues/16))
38
+ `upsertGraph` looped over `scan.symbols`, `scan.edges` and `scan.files`, and
39
+ awaited a separate `INSERT` for each row. Parsing a 639-file React Native repo
40
+ takes 1.5s and produces 50,353 rows; against a Neon instance with a 100ms
41
+ round-trip that is 84 minutes of pure network wait, and the scan never
42
+ completed. Rows are now written 500 per statement, which takes the same repo
43
+ to 20s end to end. SQLite used a prepared statement and MongoDB used
44
+ `bulkWrite` already; only Postgres wrote row by row.
45
+ - **Postgres: changing embedding dimensions permanently disabled vector search,
46
+ and `reembed` could not repair it.**
47
+ ([#14](https://github.com/Avijit07x/claude-db/issues/14))
48
+ Installing `@xenova/transformers`, which `doctor` recommends, moves `auto`
49
+ from the 256d builtin embedder to 384d MiniLM. `ensureVectorColumn` only ever
50
+ created the `embedding` column, never widened an existing one, so every write
51
+ was rejected for width and `searchVector` returned nothing on every query.
52
+ Rows were still stamped with the new `embedder`, and `reembed` skipped any row
53
+ whose label already matched, so the corpus could not recover: on one 455-row
54
+ database, 291 rows kept stale 256d vectors, 164 had none, and search silently
55
+ ran keyword-only for weeks. `reembed` now rebuilds the column at the new width
56
+ and re-embeds every row, `embedder` is only written when the vector is stored,
57
+ and rows missing a vector are treated as stale. The Postgres adapter also
58
+ hydrates `embedding` on read, as SQLite and MongoDB already did.
59
+
60
+ ### Added
61
+
62
+ - **Rendering a component in `.tsx` now produces a graph edge.**
63
+ ([#19](https://github.com/Avijit07x/claude-db/issues/19))
64
+ The TSX spec was TypeScript with a different label, and JSX has no rule, so
65
+ `<Card />` matched nothing: a presentational component had no inbound edges at
66
+ all and `find_usages` reported it as unused. `jsx_opening_element` and
67
+ `jsx_self_closing_element` are now reference rules, filtered to capitalised
68
+ names via a new optional `namePattern` on `ReferenceRule` so host elements
69
+ like `div` and `span` are not mistaken for components. The same rules apply to
70
+ `.jsx`, which resolves to the JavaScript spec. On a 639-file React
71
+ Native repo this adds 7,904 edges, 3,477 of them resolving to a symbol in the
72
+ repo, for 0.2s of extra parse time.
73
+
3
74
  ## 0.8.3
4
75
 
5
76
  ### Fixed
@@ -12,7 +83,7 @@
12
83
  `/home/f/work/epiclin_2` was looked up under `-home-f-work-epiclin_2` while
13
84
  its transcripts sat in `-home-f-work-epiclin-2`, so `flush`, `adoption` and
14
85
  `setup` all reported nothing to ingest. The name is now derived the same way
15
- Claude Code writes it — one dash per non-alphanumeric character — which also
86
+ Claude Code writes it, one dash per non-alphanumeric character, which also
16
87
  covers paths with spaces, `@`, `~` or accented characters.
17
88
 
18
89
  ## 0.8.2
@@ -108,11 +108,15 @@ export async function cmdReembed(argv = []) {
108
108
  console.error('No embedder available; nothing to do.');
109
109
  process.exit(1);
110
110
  }
111
+ let migrated = false;
112
+ if (!scoped && ctx.store.migrateVectorDims) {
113
+ migrated = await ctx.store.migrateVectorDims(embedder.dimensions);
114
+ }
111
115
  let updated = 0;
112
116
  let skipped = 0;
113
117
  const filter = scoped ? { project: resolveProject(undefined) } : {};
114
118
  const scanned = await eachObservation(ctx, filter, async (batch) => {
115
- const stale = batch.filter((obs) => obs.embedder !== embedder.id);
119
+ const stale = batch.filter((obs) => obs.embedder !== embedder.id || !obs.embedding?.length);
116
120
  skipped += batch.length - stale.length;
117
121
  if (stale.length === 0)
118
122
  return;
@@ -122,6 +126,9 @@ export async function cmdReembed(argv = []) {
122
126
  process.stderr.write(`\r${updated} re-embedded...`);
123
127
  });
124
128
  process.stderr.write('\r');
129
+ if (migrated) {
130
+ console.log(`Rebuilt vector storage at ${embedder.dimensions}d.`);
131
+ }
125
132
  console.log(`Scanned ${scanned}, re-embedded ${updated} with ${embedder.id}` +
126
133
  `${skipped > 0 ? `, ${skipped} already current` : ''}.`);
127
134
  }
@@ -19,9 +19,24 @@ const TYPED_DEFINITIONS = [
19
19
  { kind: 'type_alias_declaration', field: ['name'], symbol: 'type' },
20
20
  { kind: 'enum_declaration', field: ['name'], symbol: 'enum' },
21
21
  ];
22
+ const TYPE_DECLARATION_SITES = [
23
+ 'interface_declaration',
24
+ 'type_alias_declaration',
25
+ 'class_declaration',
26
+ 'abstract_class_declaration',
27
+ 'enum_declaration',
28
+ 'type_parameter',
29
+ 'implements_clause',
30
+ ];
22
31
  const TYPED_REFERENCES = [
23
32
  { kind: 'extends_clause', field: ['value'], relation: 'extends' },
24
33
  { kind: 'implements_clause', field: [], relation: 'implements' },
34
+ {
35
+ kind: 'type_identifier',
36
+ field: [],
37
+ relation: 'references',
38
+ excludeParents: TYPE_DECLARATION_SITES,
39
+ },
25
40
  ];
26
41
  export const typescript = {
27
42
  id: 'TypeScript',
@@ -30,11 +45,27 @@ export const typescript = {
30
45
  definitions: [...SHARED_DEFINITIONS, ...TYPED_DEFINITIONS],
31
46
  references: [...SHARED_REFERENCES, ...TYPED_REFERENCES],
32
47
  };
48
+ const COMPONENT_NAME = /^[A-Z]/;
49
+ const JSX_REFERENCES = [
50
+ {
51
+ kind: 'jsx_opening_element',
52
+ field: ['name'],
53
+ relation: 'references',
54
+ namePattern: COMPONENT_NAME,
55
+ },
56
+ {
57
+ kind: 'jsx_self_closing_element',
58
+ field: ['name'],
59
+ relation: 'references',
60
+ namePattern: COMPONENT_NAME,
61
+ },
62
+ ];
33
63
  export const tsx = {
34
64
  ...typescript,
35
65
  id: 'Tsx',
36
66
  label: 'tsx',
37
67
  extensions: ['.tsx'],
68
+ references: [...SHARED_REFERENCES, ...TYPED_REFERENCES, ...JSX_REFERENCES],
38
69
  };
39
70
  export const javascript = {
40
71
  id: 'JavaScript',
@@ -44,5 +75,9 @@ export const javascript = {
44
75
  ...SHARED_DEFINITIONS,
45
76
  { kind: 'class_declaration', field: ['name'], symbol: 'class' },
46
77
  ],
47
- references: [...SHARED_REFERENCES, { kind: 'class_heritage', field: [], relation: 'extends' }],
78
+ references: [
79
+ ...SHARED_REFERENCES,
80
+ { kind: 'class_heritage', field: [], relation: 'extends' },
81
+ ...JSX_REFERENCES,
82
+ ],
48
83
  };
@@ -8,6 +8,8 @@ export interface ReferenceRule {
8
8
  kind: string;
9
9
  field: string[];
10
10
  relation: EdgeRelation;
11
+ namePattern?: RegExp;
12
+ excludeParents?: string[];
11
13
  }
12
14
  export interface LanguageSpec {
13
15
  id: string;
@@ -13,6 +13,7 @@ export interface AstNode {
13
13
  kind(): string;
14
14
  range(): AstRange;
15
15
  field(name: string): AstNode | null;
16
+ parent(): AstNode | null;
16
17
  findAll(matcher: {
17
18
  rule: {
18
19
  kind: string;
@@ -12,5 +12,5 @@ export interface Extraction {
12
12
  symbols: CodeSymbol[];
13
13
  references: Reference[];
14
14
  }
15
- export declare function symbolId(project: string, file: string, name: string, kind: string): string;
15
+ export declare function symbolId(project: string, file: string, name: string, kind: string, index?: number): string;
16
16
  export declare function extractFile(file: SourceFile, project: string): Extraction;
@@ -4,16 +4,27 @@ import { callsIn, declarationsIn } from '../languages/index.js';
4
4
  import { languageHandle, loadParser } from '../parser.js';
5
5
  const CALLABLE = new Set(['function', 'method', 'class']);
6
6
  const IDENTIFIER = /^[\w$]+[?!]?$/;
7
- export function symbolId(project, file, name, kind) {
8
- return observationId('graph', 0, `${project}\0${file}\0${name}\0${kind}`);
7
+ export function symbolId(project, file, name, kind, index = 0) {
8
+ const seed = `${project}\0${file}\0${name}\0${kind}`;
9
+ return observationId('graph', 0, index === 0 ? seed : `${seed}\0${index}`);
10
+ }
11
+ function counter() {
12
+ const seen = new Map();
13
+ return (name, kind) => {
14
+ const key = `${name}\0${kind}`;
15
+ const index = seen.get(key) ?? 0;
16
+ seen.set(key, index + 1);
17
+ return index;
18
+ };
9
19
  }
10
20
  function extractByPattern(file, project) {
11
21
  const lines = file.source.split('\n');
12
22
  const symbols = [];
13
23
  const spans = [];
24
+ const occurrence = counter();
14
25
  for (const declaration of declarationsIn(file.source)) {
15
26
  const symbol = {
16
- id: symbolId(project, file.path, declaration.name, declaration.kind),
27
+ id: symbolId(project, file.path, declaration.name, declaration.kind, occurrence(declaration.name, declaration.kind)),
17
28
  project,
18
29
  name: declaration.name,
19
30
  kind: declaration.kind,
@@ -84,6 +95,7 @@ export function extractFile(file, project) {
84
95
  const lines = file.source.split('\n');
85
96
  const symbols = [];
86
97
  const spans = [];
98
+ const occurrence = counter();
87
99
  for (const rule of file.spec.definitions) {
88
100
  for (const node of root.findAll({ rule: { kind: rule.kind } })) {
89
101
  const named = resolveField(node, rule.field);
@@ -92,7 +104,7 @@ export function extractFile(file, project) {
92
104
  continue;
93
105
  const line = named.range().start.line + 1;
94
106
  const symbol = {
95
- id: symbolId(project, file.path, name, rule.symbol),
107
+ id: symbolId(project, file.path, name, rule.symbol, occurrence(name, rule.symbol)),
96
108
  project,
97
109
  name,
98
110
  kind: rule.symbol,
@@ -133,6 +145,10 @@ export function extractFile(file, project) {
133
145
  continue;
134
146
  if (rule.relation === 'references' && name.includes('.'))
135
147
  continue;
148
+ if (rule.namePattern && !rule.namePattern.test(name))
149
+ continue;
150
+ if (rule.excludeParents?.includes(node.parent()?.kind() ?? ''))
151
+ continue;
136
152
  const line = target.range().start.line + 1;
137
153
  references.push({
138
154
  file: file.path,
@@ -5,7 +5,7 @@ export interface SourceFile {
5
5
  source: string;
6
6
  hash: string;
7
7
  }
8
- export declare const SCAN_VERSION = 3;
8
+ export declare const SCAN_VERSION = 4;
9
9
  export declare function hashOf(bytes: Buffer): string;
10
10
  export declare function listFiles(root: string): string[];
11
11
  export declare function sourceFiles(root: string): string[];
@@ -5,9 +5,9 @@ import { join } from 'node:path';
5
5
  import { BASIC_FINGERPRINT, LANGUAGES, languageFor } from '../languages/index.js';
6
6
  const MAX_BUFFER = 64 * 1024 * 1024;
7
7
  const MAX_FILE_BYTES = 1024 * 1024;
8
- export const SCAN_VERSION = 3;
8
+ export const SCAN_VERSION = 4;
9
9
  const RULES_FINGERPRINT = createHash('sha256')
10
- .update(JSON.stringify(LANGUAGES))
10
+ .update(JSON.stringify(LANGUAGES, (_key, value) => (value instanceof RegExp ? String(value) : value)))
11
11
  .update(BASIC_FINGERPRINT)
12
12
  .digest('hex')
13
13
  .slice(0, 12);
@@ -15,6 +15,7 @@ export interface MemoryStore {
15
15
  clearSummary(id: string): Promise<boolean>;
16
16
  insertObservations(observations: Observation[]): Promise<void>;
17
17
  getObservations(ids: string[]): Promise<Observation[]>;
18
+ migrateVectorDims?(dims: number): Promise<boolean>;
18
19
  remove(filter: RemoveFilter): Promise<number>;
19
20
  list(filter: ListFilter): Promise<Observation[]>;
20
21
  listProjects(): Promise<ProjectSummary[]>;
@@ -1,43 +1,63 @@
1
1
  import { toEdge, toSymbol } from './rows.js';
2
+ const CHUNK = 500;
3
+ function lastById(rows, key) {
4
+ const byId = new Map();
5
+ for (const row of rows)
6
+ byId.set(key(row), row);
7
+ return [...byId.values()];
8
+ }
9
+ function placeholders(count, width) {
10
+ const groups = [];
11
+ for (let row = 0; row < count; row += 1) {
12
+ const holes = [];
13
+ for (let col = 1; col <= width; col += 1)
14
+ holes.push(`$${row * width + col}`);
15
+ groups.push(`(${holes.join(', ')})`);
16
+ }
17
+ return groups.join(', ');
18
+ }
19
+ async function upsertChunked(client, rows, width, sql, columns) {
20
+ for (let start = 0; start < rows.length; start += CHUNK) {
21
+ const slice = rows.slice(start, start + CHUNK);
22
+ const values = [];
23
+ for (const row of slice)
24
+ values.push(...columns(row));
25
+ await client.query(sql(placeholders(slice.length, width)), values);
26
+ }
27
+ }
2
28
  export async function upsertGraph(pool, scan) {
3
29
  const client = await pool.connect();
4
30
  try {
5
31
  await client.query('BEGIN');
6
- for (const s of scan.symbols) {
7
- await client.query(`INSERT INTO symbols (id, project, name, kind, file, line, lang, signature)
8
- VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
32
+ await upsertChunked(client, lastById(scan.symbols, (s) => s.id), 8, (values) => `INSERT INTO symbols (id, project, name, kind, file, line, lang, signature)
33
+ VALUES ${values}
9
34
  ON CONFLICT (id) DO UPDATE SET
10
35
  name = EXCLUDED.name, kind = EXCLUDED.kind, file = EXCLUDED.file,
11
36
  line = EXCLUDED.line, lang = EXCLUDED.lang,
12
- signature = EXCLUDED.signature`, [s.id, s.project, s.name, s.kind, s.file, s.line, s.lang, s.signature]);
13
- }
14
- for (const e of scan.edges) {
15
- await client.query(`INSERT INTO symbol_edges (id, project, src_id, src_name, dst_id, dst_name,
37
+ signature = EXCLUDED.signature`, (s) => [s.id, s.project, s.name, s.kind, s.file, s.line, s.lang, s.signature]);
38
+ await upsertChunked(client, lastById(scan.edges, (e) => e.id), 11, (values) => `INSERT INTO symbol_edges (id, project, src_id, src_name, dst_id, dst_name,
16
39
  relation, confidence, score, file, line)
17
- VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
40
+ VALUES ${values}
18
41
  ON CONFLICT (id) DO UPDATE SET
19
42
  dst_id = EXCLUDED.dst_id, dst_name = EXCLUDED.dst_name,
20
43
  relation = EXCLUDED.relation, confidence = EXCLUDED.confidence,
21
- score = EXCLUDED.score, file = EXCLUDED.file, line = EXCLUDED.line`, [
22
- e.id,
23
- e.project,
24
- e.srcId,
25
- e.srcName,
26
- e.dstId,
27
- e.dstName,
28
- e.relation,
29
- e.confidence,
30
- e.score,
31
- e.file,
32
- e.line,
33
- ]);
34
- }
35
- for (const f of scan.files) {
36
- await client.query(`INSERT INTO scanned_files (project, path, hash, scanned_at)
37
- VALUES ($1, $2, $3, $4)
44
+ score = EXCLUDED.score, file = EXCLUDED.file, line = EXCLUDED.line`, (e) => [
45
+ e.id,
46
+ e.project,
47
+ e.srcId,
48
+ e.srcName,
49
+ e.dstId,
50
+ e.dstName,
51
+ e.relation,
52
+ e.confidence,
53
+ e.score,
54
+ e.file,
55
+ e.line,
56
+ ]);
57
+ await upsertChunked(client, lastById(scan.files, (f) => `${f.project}\u0000${f.path}`), 4, (values) => `INSERT INTO scanned_files (project, path, hash, scanned_at)
58
+ VALUES ${values}
38
59
  ON CONFLICT (project, path) DO UPDATE SET
39
- hash = EXCLUDED.hash, scanned_at = EXCLUDED.scanned_at`, [f.project, f.path, f.hash, f.scannedAt]);
40
- }
60
+ hash = EXCLUDED.hash, scanned_at = EXCLUDED.scanned_at`, (f) => [f.project, f.path, f.hash, f.scannedAt]);
41
61
  await client.query('COMMIT');
42
62
  }
43
63
  catch (error) {
@@ -17,6 +17,7 @@ export declare class PostgresStore implements MemoryStore {
17
17
  writeMeta(): Promise<void>;
18
18
  ensureTagsIndexed(): Promise<void>;
19
19
  readVectorDims(): Promise<number | null>;
20
+ migrateVectorDims(dims: number): Promise<boolean>;
20
21
  private ensureVectorColumn;
21
22
  close(): Promise<void>;
22
23
  ping(): Promise<boolean>;
@@ -47,6 +47,23 @@ export class PostgresStore {
47
47
  async readVectorDims() {
48
48
  return metaOps.readVectorDims(this.pool);
49
49
  }
50
+ async migrateVectorDims(dims) {
51
+ if (!this.vectorEnabled)
52
+ return false;
53
+ if (!Number.isInteger(dims) || dims < 1 || dims > 16000)
54
+ return false;
55
+ if (this.vectorDims === dims)
56
+ return false;
57
+ await this.pool.query('ALTER TABLE observations DROP COLUMN IF EXISTS embedding');
58
+ await this.pool.query(`ALTER TABLE observations ADD COLUMN embedding vector(${dims})`);
59
+ await this.pool.query('UPDATE observations SET embedder = NULL');
60
+ await this.pool.query(`CREATE INDEX IF NOT EXISTS idx_obs_embedding
61
+ ON observations USING hnsw (embedding vector_cosine_ops)`);
62
+ this.vectorDims = await this.readVectorDims();
63
+ this.warnedDims = false;
64
+ await this.writeMeta();
65
+ return true;
66
+ }
50
67
  async ensureVectorColumn(dims) {
51
68
  if (!this.vectorEnabled)
52
69
  return false;
@@ -7,6 +7,7 @@ export async function insertObservations(pool, ensureVectorColumn, observations)
7
7
  try {
8
8
  await client.query('BEGIN');
9
9
  for (const obs of observations) {
10
+ const vectorStored = vectors && obs.embedding?.length === width;
10
11
  const columns = [
11
12
  'id',
12
13
  'session_id',
@@ -31,11 +32,11 @@ export async function insertObservations(pool, ensureVectorColumn, observations)
31
32
  JSON.stringify(obs.files),
32
33
  JSON.stringify(obs.tags),
33
34
  obs.createdAt,
34
- obs.embedder ?? null,
35
+ vectorStored ? (obs.embedder ?? null) : null,
35
36
  obs.author ?? null,
36
37
  obs.status ?? 'done',
37
38
  ];
38
- if (vectors && obs.embedding?.length === width) {
39
+ if (vectorStored && obs.embedding) {
39
40
  columns.push('embedding');
40
41
  values.push(`[${obs.embedding.join(',')}]`);
41
42
  }
@@ -11,6 +11,19 @@ export function toSession(row) {
11
11
  session.summary = row['summary'];
12
12
  return session;
13
13
  }
14
+ function toVector(value) {
15
+ if (Array.isArray(value))
16
+ return value;
17
+ if (typeof value !== 'string')
18
+ return null;
19
+ const trimmed = value.trim();
20
+ if (!trimmed.startsWith('[') || !trimmed.endsWith(']'))
21
+ return null;
22
+ const inner = trimmed.slice(1, -1);
23
+ if (inner.length === 0)
24
+ return null;
25
+ return inner.split(',').map(Number);
26
+ }
14
27
  export function toObservation(row) {
15
28
  const obs = {
16
29
  id: row['id'],
@@ -23,6 +36,9 @@ export function toObservation(row) {
23
36
  tags: row['tags'] ?? [],
24
37
  createdAt: Number(row['created_at']),
25
38
  };
39
+ const embedding = toVector(row['embedding']);
40
+ if (embedding)
41
+ obs.embedding = embedding;
26
42
  if (row['embedder'] != null)
27
43
  obs.embedder = row['embedder'];
28
44
  if (row['author'] != null)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "claude-db",
3
- "version": "0.8.3",
3
+ "version": "0.8.5",
4
4
  "description": "Persistent memory for Claude Code. Captures why decisions were made, not just which files changed. Bring your own database: SQLite, MongoDB or Postgres. No cloud, no subscription.",
5
5
  "license": "Apache-2.0",
6
6
  "type": "module",