@allansantos-dev/smart-tool 0.9.77 → 0.9.79
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/README.md +3 -3
- package/affected_tests.py +4 -0
- package/code_graph.py +39 -17
- package/code_graph_go.py +6 -6
- package/code_graph_java.mjs +10 -9
- package/code_graph_js.cjs +8 -8
- package/code_impact.py +3 -1
- package/disk_cleanup.py +64 -12
- package/doc_scope.py +1 -1
- package/duplicates.py +1 -1
- package/index_scope.py +165 -6
- package/layers.py +3 -3
- package/package.json +1 -1
- package/pr_impact.py +2 -0
- package/project_admission.py +37 -4
- package/project_monitor.py +1 -0
- package/smart_tool_daemon.py +4 -4
- package/version.py +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,28 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.9.79 - beta
|
|
4
|
+
|
|
5
|
+
- A graph cut by its limits says what it left out: `left_out` with the files, symbols and call or import relations
|
|
6
|
+
dropped, and a sentence in the note, in the graph answers (`project_manage action=graph`), `layers`, `docs generate`,
|
|
7
|
+
`duplicates`, the affected tests and the PR impact. Files skipped by the 24 MB analysis text limit count with the
|
|
8
|
+
files over the file limit. The graph cache is rebuilt once. The limits stay as they are: doubling them took the
|
|
9
|
+
build of a 6,133-file project from 23 s to 57 s and its memory from 0.7 GB to 1.4 GB
|
|
10
|
+
(`docs/research/2026-11-15-tetos-do-grafo-medidos.md`). The gateway rerank gain over the hybrid order was measured on
|
|
11
|
+
the gold set (first place 35 to 39 of 43, MRR@5 0.868 to 0.919, 1.4 s median per search), so rerank stays optional
|
|
12
|
+
(`docs/research/2026-11-15-ganho-do-rerank-medido.md`). Tests: `tests/test_code_graph_limits.py`,
|
|
13
|
+
`tests/test_affected_tests.py`, `tests/test_pr_impact.py`.
|
|
14
|
+
|
|
15
|
+
## 0.9.78 - beta
|
|
16
|
+
|
|
17
|
+
- A registered project whose folder no longer exists is junk: `project_manage action=junk` lists it and the junk
|
|
18
|
+
path (confirm, and the cleanup at daemon start) removes its index files, never any source file. The manifest
|
|
19
|
+
(`junk-manifests/junk-<time>.json`) is written before deleting and records the project, its registration settings,
|
|
20
|
+
counts, scope per view and the bytes freed. A folder is gone only when its volume root exists, so a project on a
|
|
21
|
+
drive or share that is not connected is kept. Tests: `tests/test_project_admission.py`.
|
|
22
|
+
- The index scope skips Python virtual environments (a folder holding `pyvenv.cfg` or with `venv` in its name) at any
|
|
23
|
+
depth and honors the `.gitignore` files inside the project; a folder named in the scope is still indexed. A new
|
|
24
|
+
project under the system temporary folder is refused with the reason. Tests: `tests/test_scope_rules.py`.
|
|
25
|
+
|
|
3
26
|
## 0.9.77 - beta
|
|
4
27
|
|
|
5
28
|
- The Claude Code hooks Smart Tool writes name their owner and the file that holds their port: the two HTTP hook URLs
|
package/README.md
CHANGED
|
@@ -17,7 +17,7 @@ registered as the `playwright` server where this machine lets a browser be drive
|
|
|
17
17
|
An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
|
|
18
18
|
are cheaper, or just tells the agent what Smart Tool would do.
|
|
19
19
|
|
|
20
|
-
**Status:** 0.9.
|
|
20
|
+
**Status:** 0.9.79 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
|
|
21
21
|
|
|
22
22
|
## How it works
|
|
23
23
|
|
|
@@ -83,7 +83,7 @@ gateway such as Ollama, nothing about your code leaves the machine.
|
|
|
83
83
|
## Install
|
|
84
84
|
|
|
85
85
|
```powershell
|
|
86
|
-
npx @allansantos-dev/smart-tool@0.9.
|
|
86
|
+
npx @allansantos-dev/smart-tool@0.9.79 install
|
|
87
87
|
```
|
|
88
88
|
|
|
89
89
|
Updates install themselves, on by default: the tray checks npm when it starts (at logon) and every 24 h, and when there
|
|
@@ -390,7 +390,7 @@ fails to load is listed in the setup screen with its error and does not affect t
|
|
|
390
390
|
## Uninstall
|
|
391
391
|
|
|
392
392
|
```powershell
|
|
393
|
-
npx @allansantos-dev/smart-tool@0.9.
|
|
393
|
+
npx @allansantos-dev/smart-tool@0.9.79 uninstall [--target DIR] [--keep-data]
|
|
394
394
|
```
|
|
395
395
|
|
|
396
396
|
It asks nothing and opens no window. It stops the tray and the daemon, removes the logon task, the hook entries of Smart
|
package/affected_tests.py
CHANGED
|
@@ -413,6 +413,10 @@ def affected(root, base="HEAD", limit=30, view_id=None):
|
|
|
413
413
|
"view": (data.get("selected") or {}).get("label"),
|
|
414
414
|
"note": NOTE,
|
|
415
415
|
}
|
|
416
|
+
if data.get("truncated"):
|
|
417
|
+
result["truncated"] = True
|
|
418
|
+
result["left_out"] = data["left_out"]
|
|
419
|
+
result["note"] += " " + code_graph.left_out_sentence(data) + " Tests beyond the cut are not listed."
|
|
416
420
|
if len(ordered) > limit:
|
|
417
421
|
result["omitted"] = {"tests": len(ordered) - limit}
|
|
418
422
|
if not changes:
|
package/code_graph.py
CHANGED
|
@@ -23,7 +23,7 @@ import web_document_graph
|
|
|
23
23
|
import code_graph_go
|
|
24
24
|
import document_text
|
|
25
25
|
|
|
26
|
-
VERSION =
|
|
26
|
+
VERSION = 6
|
|
27
27
|
SOURCE_EXTENSIONS = ('.py','.go','.mod','.java','.js','.jsx','.ts','.tsx','.mjs','.cjs','.mts','.cts','.json','.html','.htm','.css','.scss','.sass','.less','.md','.markdown')
|
|
28
28
|
MAX_OVERLAY_BYTES = 5 * 1024 * 1024
|
|
29
29
|
MAX_FILES = 3000
|
|
@@ -71,9 +71,10 @@ def _snapshot(path):
|
|
|
71
71
|
columns = {row[1] for row in conn.execute('PRAGMA table_info(manifest)')}
|
|
72
72
|
size = 'm.size' if 'size' in columns else 'NULL'
|
|
73
73
|
mtime = 'm.mtime_ns' if 'mtime_ns' in columns else 'NULL'
|
|
74
|
-
records = conn.execute(f'SELECT m.path,m.hash,{size},{mtime},count(c.id),max(c.end_line) FROM manifest m LEFT JOIN chunks c ON c.path=m.path GROUP BY m.path ORDER BY m.path LIMIT ?', (MAX_FILES
|
|
75
|
-
|
|
76
|
-
|
|
74
|
+
records = conn.execute(f'SELECT m.path,m.hash,{size},{mtime},count(c.id),max(c.end_line) FROM manifest m LEFT JOIN chunks c ON c.path=m.path GROUP BY m.path ORDER BY m.path LIMIT ?', (MAX_FILES,)).fetchall()
|
|
75
|
+
total = conn.execute('SELECT count(*) FROM manifest').fetchone()[0]
|
|
76
|
+
files, sources, diagnostics, used, over_text = [], {}, [], 0, 0
|
|
77
|
+
for original, digest, byte_size, stamp, chunks, line_count in records:
|
|
77
78
|
file_path = original.replace('\\','/')
|
|
78
79
|
record = {'id':file_path,'path':file_path,'hash':digest,'size':byte_size,'mtime_ns':stamp,
|
|
79
80
|
'chunks':chunks,'lines':line_count or 0,'group':posixpath.dirname(file_path) or '(root)',
|
|
@@ -87,10 +88,11 @@ def _snapshot(path):
|
|
|
87
88
|
diagnostics.append({'path':file_path,'reason':error})
|
|
88
89
|
elif used+len(text) > MAX_TEXT:
|
|
89
90
|
diagnostics.append({'path':file_path,'reason':'Content limit for this analysis reached.'})
|
|
91
|
+
over_text += 1
|
|
90
92
|
else:
|
|
91
93
|
used += len(text)
|
|
92
94
|
sources[file_path] = text
|
|
93
|
-
return files, sources, diagnostics,
|
|
95
|
+
return files, sources, diagnostics, max(total-MAX_FILES, 0) + over_text
|
|
94
96
|
finally:
|
|
95
97
|
conn.close()
|
|
96
98
|
|
|
@@ -247,7 +249,8 @@ def _python_graph(sources):
|
|
|
247
249
|
visit_scope(tree)
|
|
248
250
|
if used_module:symbols.append(module)
|
|
249
251
|
return {'symbols':symbols,'dependencies':dependencies[:MAX_EDGES],'calls':calls,'unresolved':unresolved,
|
|
250
|
-
'diagnostics':diagnostics,'parsed':list(trees),
|
|
252
|
+
'diagnostics':diagnostics,'parsed':list(trees),
|
|
253
|
+
'left_out':{'symbols':dropped['symbols'],'edges':dropped['calls']+max(len(dependencies)-MAX_EDGES,0)}}
|
|
251
254
|
|
|
252
255
|
|
|
253
256
|
NODE_HEAP_MB = (1024, 4096)
|
|
@@ -298,15 +301,22 @@ def _analyze(path):
|
|
|
298
301
|
return _analyze_sources(*_snapshot(path))
|
|
299
302
|
|
|
300
303
|
|
|
301
|
-
def _analyze_sources(files,sources,diagnostics,
|
|
304
|
+
def _analyze_sources(files,sources,diagnostics,files_left_out):
|
|
305
|
+
"""Analyzes the sources of one snapshot. `left_out` counts what the limits cut: files the snapshot never read
|
|
306
|
+
(files_left_out), symbols and call or import occurrences each analyzer dropped at its own cap, and what the merged
|
|
307
|
+
graph drops at the global caps (symbols past MAX_SYMBOLS, calls whose endpoint was dropped, relations past MAX_EDGES).
|
|
308
|
+
The counts are lower bounds: what lies inside left-out files, and the calls made inside functions an analyzer
|
|
309
|
+
dropped, are never seen and not counted."""
|
|
302
310
|
merged={'symbols':[],'dependencies':[],'calls':[],'unresolved':[],'diagnostics':diagnostics,'parsed':[]}
|
|
311
|
+
left_out={'files':files_left_out,'symbols':0,'edges':0}
|
|
303
312
|
retryable=False
|
|
304
313
|
for output in (_python_graph(sources),_javascript_graph(sources,[f['path'] for f in files]),_java_graph(sources),code_graph_go.analyze(sources,ANALYZER_LIMITS),web_document_graph.analyze(sources,[f['path'] for f in files])):
|
|
305
314
|
for key in merged:merged[key].extend(output.get(key,[]))
|
|
306
|
-
|
|
315
|
+
for kind,count in output.get('left_out',{}).items():left_out[kind]+=count
|
|
307
316
|
retryable |= output.get('retryable',False)
|
|
308
|
-
|
|
309
|
-
|
|
317
|
+
kept_symbols=merged['symbols'][:MAX_SYMBOLS]
|
|
318
|
+
left_out['symbols']+=len(merged['symbols'])-len(kept_symbols)
|
|
319
|
+
by_id={sym['id']:sym for sym in kept_symbols}
|
|
310
320
|
# Relações repetidas preservam contagem e locais de chamada sem poluir o desenho.
|
|
311
321
|
def collapse(rows,fields):
|
|
312
322
|
result={}
|
|
@@ -316,14 +326,17 @@ def _analyze_sources(files,sources,diagnostics,truncated):
|
|
|
316
326
|
result[key]['count']+=1
|
|
317
327
|
if len(result[key]['lines'])<20:result[key]['lines'].append(row.get('line'))
|
|
318
328
|
return list(result.values())
|
|
329
|
+
linked=[r for r in merged['calls'] if r['source'] in by_id and r['target'] in by_id]
|
|
330
|
+
left_out['edges']+=len(merged['calls'])-len(linked)
|
|
319
331
|
deps=collapse(merged['dependencies'],('source','target','specifier','kind'))
|
|
320
|
-
calls=collapse(
|
|
332
|
+
calls=collapse(linked,('source','target','kind'))
|
|
333
|
+
left_out['edges']+=sum(r['count'] for r in deps[MAX_EDGES:])+sum(r['count'] for r in calls[MAX_EDGES:])
|
|
321
334
|
supported=set(merged['parsed'])
|
|
322
335
|
for file in files:file['analysis']='parsed' if file['path'] in supported else 'not_analyzed'
|
|
323
|
-
return {'files':files,'symbols':
|
|
336
|
+
return {'files':files,'symbols':kept_symbols,
|
|
324
337
|
'dependencies':deps[:MAX_EDGES],'calls':calls[:MAX_EDGES],
|
|
325
338
|
'unresolved':merged['unresolved'][:1000],'diagnostics':merged['diagnostics'],
|
|
326
|
-
'truncated':
|
|
339
|
+
'truncated':any(left_out.values()),'left_out':left_out,'retryable':retryable,
|
|
327
340
|
'counts':{'files':len(files),'parsed_files':len(supported),'symbols':len(merged['symbols']),
|
|
328
341
|
'resolved_imports':sum(bool(r['target']) for r in deps),
|
|
329
342
|
'external_imports':sum(r['resolution']=='external' for r in deps),
|
|
@@ -353,13 +366,21 @@ def _focus(data,file_path):
|
|
|
353
366
|
return data
|
|
354
367
|
|
|
355
368
|
|
|
369
|
+
def left_out_sentence(data):
|
|
370
|
+
"""One sentence with how much the analysis limits cut from the graph, or None when nothing was left out."""
|
|
371
|
+
if not data.get('truncated'):
|
|
372
|
+
return None
|
|
373
|
+
left=data['left_out']
|
|
374
|
+
return (f"Graph cut by analysis limits: {left['files']:,} files, {left['symbols']:,} symbols and {left['edges']:,} call and import relations left out"
|
|
375
|
+
+' (lower bounds: what lies inside left-out files and functions is not counted).')
|
|
376
|
+
|
|
377
|
+
|
|
356
378
|
def _by_file(data):
|
|
357
379
|
by_file={}
|
|
358
380
|
for symbol in data['symbols']:
|
|
359
381
|
by_file.setdefault(symbol['path'],[]).append(symbol)
|
|
360
382
|
parsed={f['path'] for f in data['files'] if f.get('analysis')=='parsed'}
|
|
361
|
-
|
|
362
|
-
return {'symbols':by_file,'parsed':parsed,'note':note}
|
|
383
|
+
return {'symbols':by_file,'parsed':parsed,'note':left_out_sentence(data)}
|
|
363
384
|
|
|
364
385
|
|
|
365
386
|
def _disk_path(view_path):
|
|
@@ -493,6 +514,7 @@ def build(root,view_id=None,storage_id=None,file_path=None,background=False):
|
|
|
493
514
|
file['status']='indexed' if (current.st_mtime_ns,current.st_size)==(file['mtime_ns'],file['size']) else 'pending'
|
|
494
515
|
except OSError:file['status']='missing'
|
|
495
516
|
file.pop('mtime_ns',None)
|
|
517
|
+
if data['truncated']:data['note']=left_out_sentence(data)
|
|
496
518
|
data.update(views=inventory['views'],selected=inventory['selected'],current_view=inventory['current_view'],
|
|
497
519
|
message=inventory['message'],read_only=True,source='indexed_snapshot',generated_at=time.time(),
|
|
498
520
|
limits={'files':MAX_FILES,'symbols':MAX_SYMBOLS,'edges':MAX_EDGES,'display_nodes_default':120},
|
|
@@ -601,7 +623,7 @@ def build_overlay(root,changes,view_id=None,shared=False):
|
|
|
601
623
|
if not inventory['selected']:
|
|
602
624
|
return {**inventory,'symbols':[],'dependencies':[],'calls':[],'diagnostics':[],'counts':{},'files':[]}
|
|
603
625
|
path=inventory['selected']['path'];stat=os.stat(path)
|
|
604
|
-
files,sources,diagnostics,
|
|
626
|
+
files,sources,diagnostics,files_left_out=_snapshot(path)
|
|
605
627
|
by_path={f['path']:f for f in files};overlaid=[]
|
|
606
628
|
for rel,status in sorted(changes.items()):
|
|
607
629
|
full=os.path.join(root,rel)
|
|
@@ -624,7 +646,7 @@ def build_overlay(root,changes,view_id=None,shared=False):
|
|
|
624
646
|
if not _ANALYSIS_SLOTS.acquire(timeout=3):
|
|
625
647
|
raise RuntimeError('Two map analyses are in progress. Wait for them to finish and retry.')
|
|
626
648
|
try:
|
|
627
|
-
cached=_analyze_sources(list(by_path.values()),sources,diagnostics,
|
|
649
|
+
cached=_analyze_sources(list(by_path.values()),sources,diagnostics,files_left_out)
|
|
628
650
|
finally:
|
|
629
651
|
_ANALYSIS_SLOTS.release()
|
|
630
652
|
if not cached['retryable']:
|
package/code_graph_go.py
CHANGED
|
@@ -352,7 +352,7 @@ def _default_alias(spec):
|
|
|
352
352
|
|
|
353
353
|
def analyze(sources, limits=None):
|
|
354
354
|
"""Imports, symbols (functions, methods, types) and resolved calls of the Go files in sources (path -> text).
|
|
355
|
-
limits caps symbols and edges ({'symbols': n, 'edges': n});
|
|
355
|
+
limits caps symbols and edges ({'symbols': n, 'edges': n}); 'left_out' counts the symbols and calls dropped at the caps."""
|
|
356
356
|
limits = limits or {}
|
|
357
357
|
max_symbols, max_edges = limits.get("symbols", float("inf")), limits.get("edges", float("inf"))
|
|
358
358
|
go_files = {path: text for path, text in sources.items() if path.endswith(".go")}
|
|
@@ -428,7 +428,7 @@ def _symbols_and_calls(files, max_symbols, max_edges):
|
|
|
428
428
|
target = imports_of[path].get(alias)
|
|
429
429
|
return (target, name) if target else None
|
|
430
430
|
|
|
431
|
-
index, results, symbols, calls, unresolved,
|
|
431
|
+
index, results, symbols, calls, unresolved, left_out = {}, {}, [], [], [], {"symbols": 0, "edges": 0}
|
|
432
432
|
for path, info in files.items():
|
|
433
433
|
tables = index.setdefault(key_of(path), {"funcs": {}, "methods": {}, "types": {}, "fields": {},
|
|
434
434
|
"embedded": {}})
|
|
@@ -449,7 +449,7 @@ def _symbols_and_calls(files, max_symbols, max_edges):
|
|
|
449
449
|
symbol_id = f"{symbol_id}@{decl['start']}"
|
|
450
450
|
seen.add(symbol_id)
|
|
451
451
|
if len(symbols) >= max_symbols:
|
|
452
|
-
|
|
452
|
+
left_out["symbols"] += 1
|
|
453
453
|
continue
|
|
454
454
|
symbol = {"id": symbol_id, "path": path, "name": name, "kind": decl["kind"], "language": "go",
|
|
455
455
|
"start_line": decl["start"], "end_line": decl["end"]}
|
|
@@ -484,7 +484,7 @@ def _symbols_and_calls(files, max_symbols, max_edges):
|
|
|
484
484
|
calls.append({"source": decl["symbol"]["id"], "target": call["target"]["id"],
|
|
485
485
|
"line": call["line"], "kind": "reference", "resolution": "static"})
|
|
486
486
|
else:
|
|
487
|
-
|
|
487
|
+
left_out["edges"] += 1
|
|
488
488
|
continue
|
|
489
489
|
if call["target"] is None:
|
|
490
490
|
if len(unresolved) < 1000:
|
|
@@ -494,8 +494,8 @@ def _symbols_and_calls(files, max_symbols, max_edges):
|
|
|
494
494
|
calls.append({"source": decl["symbol"]["id"], "target": call["target"]["id"],
|
|
495
495
|
"line": call["line"], "kind": "call", "resolution": "static"})
|
|
496
496
|
else:
|
|
497
|
-
|
|
498
|
-
return {"symbols": symbols, "calls": calls, "unresolved": unresolved, "
|
|
497
|
+
left_out["edges"] += 1
|
|
498
|
+
return {"symbols": symbols, "calls": calls, "unresolved": unresolved, "left_out": left_out}
|
|
499
499
|
|
|
500
500
|
|
|
501
501
|
def _body_calls(path, tokens, decl, index, imported, own, results, resolve_type, member):
|
package/code_graph_java.mjs
CHANGED
|
@@ -10,15 +10,15 @@ if(nodeModule.registerHooks){const bundle=new URL('chevrotain/lib/chevrotain.mjs
|
|
|
10
10
|
const {parse}=await import(new URL('java-parser/src/index.js',modules).href);
|
|
11
11
|
const input=JSON.parse(fs.readFileSync(0,'utf8'));
|
|
12
12
|
const symbols=[],dependencies=[],calls=[],unresolved=[],diagnostics=[],units=[],classes=new Map(),methodNodes=new Map(),kids=new WeakMap();
|
|
13
|
-
const {symbols:MAX_SYMBOLS,edges:MAX_EDGES}=input.limits;
|
|
14
|
-
function room(list,max){if(list.length<max)return true;
|
|
13
|
+
const {symbols:MAX_SYMBOLS,edges:MAX_EDGES}=input.limits;const leftOut={symbols:0,edges:0};
|
|
14
|
+
function room(list,max,kind){if(list.length<max)return true;leftOut[kind]++;return false}
|
|
15
15
|
const NO_CHILDREN=Object.freeze([]);
|
|
16
16
|
function children(n){const groups=n?.children;if(!groups)return NO_CHILDREN;let list=kids.get(n);if(list)return list;list=[];for(const key in groups)for(const child of groups[key])if(child.children)list.push(child);kids.set(n,list);return list}
|
|
17
17
|
function all(n,name,stop=false){const result=[];function visit(node){if(node?.name===name){result.push(node);if(stop)return}for(const child of children(node))visit(child)}visit(n);return result}
|
|
18
18
|
function first(n,name){return all(n,name,true)[0]}
|
|
19
19
|
function identifiers(n){const out=[];function visit(node){if(node?.children)for(const [key,items] of Object.entries(node.children))for(const child of items){if(child.children)visit(child);else if(key==='Identifier')out.push(child)}}visit(n);return out.sort((a,b)=>a.startOffset-b.startOffset)}
|
|
20
20
|
function text(u,n){return n?.location?u.text.slice(n.location.startOffset,n.location.endOffset+1):''}
|
|
21
|
-
function addSymbol(u,n,name,kind){if(!room(symbols,MAX_SYMBOLS))return null;const s={id:u.path+'::'+name+'@'+n.location.startLine+':'+n.location.startOffset,path:u.path,name,kind,language:'java',start_line:n.location.startLine,end_line:n.location.endLine};symbols.push(s);return s}
|
|
21
|
+
function addSymbol(u,n,name,kind){if(!room(symbols,MAX_SYMBOLS,'symbols'))return null;const s={id:u.path+'::'+name+'@'+n.location.startLine+':'+n.location.startOffset,path:u.path,name,kind,language:'java',start_line:n.location.startLine,end_line:n.location.endLine};symbols.push(s);return s}
|
|
22
22
|
const classKinds=new Set(['normalClassDeclaration','normalInterfaceDeclaration','enumDeclaration','recordDeclaration']);
|
|
23
23
|
for(const file of input.files){try{const tree=parse(file.text),pkg=first(tree,'packageDeclaration'),unit={...file,tree,package:pkg?identifiers(pkg).map(t=>t.image).join('.'): '',imports:[],classes:[]};
|
|
24
24
|
for(const n of all(tree,'importDeclaration')){const raw=text(unit,n).replace(/^import\s+/,'').replace(/;\s*$/,'').trim(),stat=raw.startsWith('static '),spec=(stat?raw.slice(7):raw).replace(/\s+/g,'');unit.imports.push({spec,static:stat,line:n.location.startLine})}
|
|
@@ -38,18 +38,18 @@ for(const unit of units){
|
|
|
38
38
|
function walk(n,cls=null,method=null,vars=new Map(),owner=null){if(classKinds.has(n.name))cls=unit.classes.find(c=>c.node===n)||cls;if(cls&&n.name==='classBody'&&n!==cls.body)return;
|
|
39
39
|
if(methodNodes.has(n)){method=methodNodes.get(n);vars=new Map(cls.fields);collectVariables(unit,n,vars);owner=method.symbol}
|
|
40
40
|
if(n.name==='lambdaExpression'){vars=new Map(vars);for(const param of identifiers(n.children.lambdaParameters?.[0]))vars.set(param.image,null);owner=addSymbol(unit,n,(cls?.local||'')+'.lambda@'+n.location.startLine,'function')||owner}
|
|
41
|
-
if(cls&&['unannType','classType','interfaceType'].includes(n.name)){const type=resolveType(unit,text(unit,n),cls);if(type&&type.unit.path!==unit.path&&room(dependencies,MAX_EDGES))dependencies.push({source:unit.path,target:type.unit.path,specifier:type.fqn,line:n.location.startLine,kind:'java_type',resolution:'resolved'})}
|
|
41
|
+
if(cls&&['unannType','classType','interfaceType'].includes(n.name)){const type=resolveType(unit,text(unit,n),cls);if(type&&type.unit.path!==unit.path&&room(dependencies,MAX_EDGES,'edges'))dependencies.push({source:unit.path,target:type.unit.path,specifier:type.fqn,line:n.location.startLine,kind:'java_type',resolution:'resolved'})}
|
|
42
42
|
if(cls&&n.name==='primary'){
|
|
43
43
|
const prefix=n.children.primaryPrefix?.[0],prefixText=text(unit,prefix).trim(),suffixes=n.children.primarySuffix||[];let chain=prefixText,fluent=false;
|
|
44
44
|
const newMatch=/^new\s+([\w.$]+)/.exec(prefixText);let constructed=newMatch?resolveType(unit,newMatch[1],cls):null;
|
|
45
|
-
const link=(type,line)=>{if(type&&type.unit.path!==unit.path&&room(dependencies,MAX_EDGES))dependencies.push({source:unit.path,target:type.unit.path,specifier:type.fqn,line,kind:'java_type',resolution:'resolved'})};
|
|
46
|
-
if(constructed&&room(calls,MAX_EDGES))calls.push({source:(owner||cls.symbol).id,target:constructed.symbol.id,line:n.location.startLine,kind:'construct',resolution:'static'});
|
|
45
|
+
const link=(type,line)=>{if(type&&type.unit.path!==unit.path&&room(dependencies,MAX_EDGES,'edges'))dependencies.push({source:unit.path,target:type.unit.path,specifier:type.fqn,line,kind:'java_type',resolution:'resolved'})};
|
|
46
|
+
if(constructed&&room(calls,MAX_EDGES,'edges'))calls.push({source:(owner||cls.symbol).id,target:constructed.symbol.id,line:n.location.startLine,kind:'construct',resolution:'static'});
|
|
47
47
|
link(constructed,n.location.startLine);
|
|
48
48
|
const staticHead=/^([A-Z][\w$]*)\./.exec(prefixText);if(staticHead&&!vars.has(staticHead[1]))link(resolveType(unit,staticHead[1],cls),n.location.startLine);
|
|
49
49
|
for(const suffix of suffixes){const invoke=suffix.children.methodInvocationSuffix?.[0];if(!invoke){chain+=text(unit,suffix);continue}const args=invoke.children.argumentList?.[0],arity=args?(args.children.expression||[]).length:0;const match=/^(?:([\w.$]+)\.)?([\w$]+)$/.exec(chain);let target=null;
|
|
50
50
|
if(match&&!fluent){const receiver=match[1],name=match[2];let type=null;if(!receiver||receiver==='this')type=cls;else if(receiver==='super')type=resolveType(unit,cls.extends,cls);else if(receiver.startsWith('this.'))type=resolveType(unit,cls.fields.get(receiver.slice(5))||'',cls);else if(vars.has(receiver))type=resolveType(unit,vars.get(receiver)||'',cls);else{type=resolveType(unit,receiver,cls);link(type,invoke.location.startLine)}target=lookupMethod(type,name,arity);if(!target&&!receiver){const candidates=unit.imports.filter(i=>i.static&&(i.spec.endsWith('.'+name)||i.spec.endsWith('.*'))).map(i=>lookupMethod(classes.get(i.spec.split('.').slice(0,-1).join('.')),name,arity)).filter(Boolean);if(candidates.length===1)target=candidates[0]}}
|
|
51
51
|
if(constructed&&!fluent){const methodName=/\.([\w$]+)$/.exec(chain)?.[1];if(methodName)target=lookupMethod(constructed,methodName,arity)}
|
|
52
|
-
if(target&&room(calls,MAX_EDGES))calls.push({source:(owner||cls.symbol).id,target:target.id,line:invoke.location.startLine,kind:'call',resolution:'static'});
|
|
52
|
+
if(target&&room(calls,MAX_EDGES,'edges'))calls.push({source:(owner||cls.symbol).id,target:target.id,line:invoke.location.startLine,kind:'call',resolution:'static'});
|
|
53
53
|
else if(unresolved.length<1000)unresolved.push({path:unit.path,line:invoke.location.startLine,expression:chain.slice(0,140),reason:'External/dynamic type, interface without implementation, or ambiguous overload in the Java snapshot.'});
|
|
54
54
|
fluent=true;chain+='<result>';
|
|
55
55
|
}
|
|
@@ -57,5 +57,6 @@ for(const unit of units){
|
|
|
57
57
|
for(const child of children(n))walk(child,cls,method,vars,owner)
|
|
58
58
|
}walk(unit.tree)
|
|
59
59
|
}
|
|
60
|
-
const ids=new Set(symbols.map(s=>s.id));
|
|
61
|
-
|
|
60
|
+
const ids=new Set(symbols.map(s=>s.id)),linked=calls.filter(c=>ids.has(c.source)&&ids.has(c.target));
|
|
61
|
+
leftOut.edges+=calls.length-linked.length;
|
|
62
|
+
process.stdout.write(JSON.stringify({symbols,dependencies,calls:linked,unresolved,diagnostics,parsed:units.map(u=>u.path),left_out:leftOut}));
|
package/code_graph_js.cjs
CHANGED
|
@@ -66,8 +66,8 @@ const host={
|
|
|
66
66
|
const program=ts.createProgram(names,options,host),checker=program.getTypeChecker();
|
|
67
67
|
const symbols=[],dependencies=[],calls=[],unresolved=[],diagnostics=[],declarations=new Map(),moduleNodes=new Map();
|
|
68
68
|
const {symbols:maxSymbols,edges:maxEdges}=input.limits;
|
|
69
|
-
|
|
70
|
-
function room(list,max){if(list.length<max)return true;
|
|
69
|
+
const leftOut={symbols:0,edges:0};
|
|
70
|
+
function room(list,max,kind){if(list.length<max)return true;leftOut[kind]++;return false;}
|
|
71
71
|
function filePath(source){return source.fileName.replace('/snapshot/','');}
|
|
72
72
|
function line(source,pos){return source.getLineAndCharacterOfPosition(pos).line+1;}
|
|
73
73
|
function displayName(node) {
|
|
@@ -81,7 +81,7 @@ function displayName(node) {
|
|
|
81
81
|
}
|
|
82
82
|
function isFunction(node){return !!node.body && (ts.isFunctionDeclaration(node)||ts.isMethodDeclaration(node)||ts.isFunctionExpression(node)||ts.isArrowFunction(node)||ts.isConstructorDeclaration(node)||ts.isGetAccessor(node)||ts.isSetAccessor(node));}
|
|
83
83
|
function addSymbol(node,source,owner){
|
|
84
|
-
if(!room(symbols,maxSymbols))return null;
|
|
84
|
+
if(!room(symbols,maxSymbols,'symbols'))return null;
|
|
85
85
|
let name=displayName(node),parent=node.parent;
|
|
86
86
|
const classParent=parent&&ts.isPropertyDeclaration(parent)?parent.parent:parent;
|
|
87
87
|
const cls=classParent && (ts.isClassDeclaration(classParent)||ts.isClassExpression(classParent)) ? classParent.name?.getText() : null;
|
|
@@ -116,16 +116,16 @@ for(const source of parsed){
|
|
|
116
116
|
const active=declarations.get(node)||owner;
|
|
117
117
|
if((ts.isImportDeclaration(node)||ts.isExportDeclaration(node)) && node.moduleSpecifier && ts.isStringLiteral(node.moduleSpecifier)) {
|
|
118
118
|
const spec=node.moduleSpecifier.text,target=resolve(spec,source.fileName);
|
|
119
|
-
if(room(dependencies,maxEdges))dependencies.push({source:filePath(source),target:target?.replace('/snapshot/','')||null,specifier:spec,line:line(source,node.getStart(source)),kind:node.importClause?.isTypeOnly||node.isTypeOnly?'type_import':'import',resolution:target?'resolved':spec.startsWith('.')?'unresolved':'external'});
|
|
119
|
+
if(room(dependencies,maxEdges,'edges'))dependencies.push({source:filePath(source),target:target?.replace('/snapshot/','')||null,specifier:spec,line:line(source,node.getStart(source)),kind:node.importClause?.isTypeOnly||node.isTypeOnly?'type_import':'import',resolution:target?'resolved':spec.startsWith('.')?'unresolved':'external'});
|
|
120
120
|
}
|
|
121
121
|
if(ts.isCallExpression(node)||ts.isNewExpression(node)){
|
|
122
122
|
const expression=node.expression,at=line(source,node.getStart(source));
|
|
123
123
|
if(ts.isCallExpression(node) && (expression.kind===ts.SyntaxKind.ImportKeyword || ts.isIdentifier(expression)&&expression.text==='require') && node.arguments?.length && ts.isStringLiteral(node.arguments[0])){
|
|
124
124
|
const spec=node.arguments[0].text,target=resolve(spec,source.fileName);
|
|
125
|
-
if(room(dependencies,maxEdges))dependencies.push({source:filePath(source),target:target?.replace('/snapshot/','')||null,specifier:spec,line:at,kind:'dynamic_import',resolution:target?'resolved':spec.startsWith('.')?'unresolved':'external'});
|
|
125
|
+
if(room(dependencies,maxEdges,'edges'))dependencies.push({source:filePath(source),target:target?.replace('/snapshot/','')||null,specifier:spec,line:at,kind:'dynamic_import',resolution:target?'resolved':spec.startsWith('.')?'unresolved':'external'});
|
|
126
126
|
}else{
|
|
127
127
|
const target=targetFor(node);
|
|
128
|
-
if(target && room(calls,maxEdges))calls.push({source:(active||moduleSymbol(source)).id,target:target.id,line:at,kind:ts.isNewExpression(node)?'construct':'call',resolution:'static'});
|
|
128
|
+
if(target && room(calls,maxEdges,'edges'))calls.push({source:(active||moduleSymbol(source)).id,target:target.id,line:at,kind:ts.isNewExpression(node)?'construct':'call',resolution:'static'});
|
|
129
129
|
else if(unresolved.length<1000)unresolved.push({path:filePath(source),line:at,expression:expression.getText(source).slice(0,120),reason:'External or dynamic target, or no resolved implementation in the snapshot.'});
|
|
130
130
|
}
|
|
131
131
|
}
|
|
@@ -144,8 +144,8 @@ for(const source of parsed){
|
|
|
144
144
|
if(components.some(c=>c.template!=null)){
|
|
145
145
|
try{
|
|
146
146
|
const {parseTemplate}=require('./web_adapters/node/node_modules/@angular/compiler/fesm2022/compiler.mjs');
|
|
147
|
-
for(const component of components){if(component.template==null||!component.templatePath)continue;const pathName=component.templatePath.replace('/snapshot/',''),tree=parseTemplate(component.template,pathName);if(tree.errors?.length){diagnostics.push({path:pathName,reason:'Invalid Angular template or syntax not supported in this version.'});continue}templatePaths.add(pathName);const owner={id:pathName+'::'+filePath(component.source)+'::'+component.name+'.template@'+component.inlineLine,path:pathName,name:component.name+'.template',kind:'template',language:'angular',start_line:component.inlineLine,end_line:component.inlineLine+component.template.split('\n').length-1,location_kind:component.templatePath===component.source.fileName?'inline_template':'source_line'};if(!room(symbols,maxSymbols))break;symbols.push(owner);const shadow=new Set(),seen=new WeakSet();function walk(n,fn){if(!n||typeof n!=='object'||seen.has(n))return;seen.add(n);fn(n);for(const [key,value] of Object.entries(n)){if(['sourceSpan','span','startSourceSpan','endSourceSpan','keySpan','valueSpan','parent'].includes(key))continue;if(Array.isArray(value))value.forEach(x=>walk(x,fn));else if(value&&typeof value==='object')walk(value,fn)}}walk(tree.nodes,n=>{if(['Variable','Reference','LetDeclaration'].includes(n.constructor.name)&&n.name)shadow.add(n.name);if(n.constructor.name==='ForLoopBlock'&&n.item?.name)shadow.add(n.item.name)});const seen2=new WeakSet();function inspect(n){if(!n||typeof n!=='object'||seen2.has(n))return;seen2.add(n);const start=n.sourceSpan?.start,at=(typeof start==='number'?component.template.slice(0,start).split('\n').length-1:(start?.line??0))+component.inlineLine;if(n.constructor.name==='Call'||n.constructor.name==='SafeCall'){const receiver=n.receiver;if(receiver?.constructor.name==='PropertyRead'&&['ImplicitReceiver','ThisReceiver'].includes(receiver.receiver?.constructor.name)){const target=shadow.has(receiver.name)?null:component.methods.get(receiver.name);if(target&&room(calls,maxEdges))calls.push({source:owner.id,target:target.id,line:at,kind:'template_call',resolution:'static'});else if(unresolved.length<1000)unresolved.push({path:pathName,line:at,expression:receiver.name+'()',reason:'Unresolved template method, or a local variable that shadows the component.'})}}if(n.constructor.name==='Element'){const matches=components.filter(c=>c.selector===n.name);if(matches.length===1&&filePath(matches[0].source)!==pathName)dependencies.push({source:pathName,target:filePath(matches[0].source),specifier:n.name,line:at,kind:'angular_component',resolution:'resolved'})}if(n.constructor.name==='BindingPipe'){const matches=pipes.filter(p=>p.pipeName===n.name&&p.methods.has('transform'));if(matches.length===1){const pipe=matches[0];dependencies.push({source:pathName,target:filePath(pipe.source),specifier:n.name,line:at,kind:'angular_pipe',resolution:'resolved'});calls.push({source:owner.id,target:pipe.methods.get('transform').id,line:at,kind:'template_pipe',resolution:'static'})}}for(const [key,value] of Object.entries(n)){if(['sourceSpan','span','startSourceSpan','endSourceSpan','keySpan','valueSpan','parent'].includes(key))continue;if(Array.isArray(value))value.forEach(inspect);else if(value&&typeof value==='object')inspect(value)}}inspect(tree.nodes)
|
|
147
|
+
for(const component of components){if(component.template==null||!component.templatePath)continue;const pathName=component.templatePath.replace('/snapshot/',''),tree=parseTemplate(component.template,pathName);if(tree.errors?.length){diagnostics.push({path:pathName,reason:'Invalid Angular template or syntax not supported in this version.'});continue}templatePaths.add(pathName);const owner={id:pathName+'::'+filePath(component.source)+'::'+component.name+'.template@'+component.inlineLine,path:pathName,name:component.name+'.template',kind:'template',language:'angular',start_line:component.inlineLine,end_line:component.inlineLine+component.template.split('\n').length-1,location_kind:component.templatePath===component.source.fileName?'inline_template':'source_line'};if(!room(symbols,maxSymbols,'symbols'))break;symbols.push(owner);const shadow=new Set(),seen=new WeakSet();function walk(n,fn){if(!n||typeof n!=='object'||seen.has(n))return;seen.add(n);fn(n);for(const [key,value] of Object.entries(n)){if(['sourceSpan','span','startSourceSpan','endSourceSpan','keySpan','valueSpan','parent'].includes(key))continue;if(Array.isArray(value))value.forEach(x=>walk(x,fn));else if(value&&typeof value==='object')walk(value,fn)}}walk(tree.nodes,n=>{if(['Variable','Reference','LetDeclaration'].includes(n.constructor.name)&&n.name)shadow.add(n.name);if(n.constructor.name==='ForLoopBlock'&&n.item?.name)shadow.add(n.item.name)});const seen2=new WeakSet();function inspect(n){if(!n||typeof n!=='object'||seen2.has(n))return;seen2.add(n);const start=n.sourceSpan?.start,at=(typeof start==='number'?component.template.slice(0,start).split('\n').length-1:(start?.line??0))+component.inlineLine;if(n.constructor.name==='Call'||n.constructor.name==='SafeCall'){const receiver=n.receiver;if(receiver?.constructor.name==='PropertyRead'&&['ImplicitReceiver','ThisReceiver'].includes(receiver.receiver?.constructor.name)){const target=shadow.has(receiver.name)?null:component.methods.get(receiver.name);if(target&&room(calls,maxEdges,'edges'))calls.push({source:owner.id,target:target.id,line:at,kind:'template_call',resolution:'static'});else if(unresolved.length<1000)unresolved.push({path:pathName,line:at,expression:receiver.name+'()',reason:'Unresolved template method, or a local variable that shadows the component.'})}}if(n.constructor.name==='Element'){const matches=components.filter(c=>c.selector===n.name);if(matches.length===1&&filePath(matches[0].source)!==pathName)dependencies.push({source:pathName,target:filePath(matches[0].source),specifier:n.name,line:at,kind:'angular_component',resolution:'resolved'})}if(n.constructor.name==='BindingPipe'){const matches=pipes.filter(p=>p.pipeName===n.name&&p.methods.has('transform'));if(matches.length===1){const pipe=matches[0];dependencies.push({source:pathName,target:filePath(pipe.source),specifier:n.name,line:at,kind:'angular_pipe',resolution:'resolved'});calls.push({source:owner.id,target:pipe.methods.get('transform').id,line:at,kind:'template_pipe',resolution:'static'})}}for(const [key,value] of Object.entries(n)){if(['sourceSpan','span','startSourceSpan','endSourceSpan','keySpan','valueSpan','parent'].includes(key))continue;if(Array.isArray(value))value.forEach(inspect);else if(value&&typeof value==='object')inspect(value)}}inspect(tree.nodes)
|
|
148
148
|
}
|
|
149
149
|
}catch{diagnostics.push({path:'Angular',reason:'The Angular template analyzer did not finish. Check the local runtime.'})}
|
|
150
150
|
}
|
|
151
|
-
process.stdout.write(JSON.stringify({symbols,dependencies
|
|
151
|
+
process.stdout.write(JSON.stringify({symbols,dependencies,calls,unresolved,diagnostics,parsed:[...parsed.map(filePath),...templatePaths],left_out:leftOut}));
|
package/code_impact.py
CHANGED
|
@@ -143,7 +143,9 @@ def impact(root, symbol, view_id=None, depth=3, limit=30):
|
|
|
143
143
|
"diagnostics": diagnostics,
|
|
144
144
|
"view": (data.get("selected") or {}).get("label"),
|
|
145
145
|
"note": NOTE + (f" {len(found)} definitions share this name; each entry says which one ('of'); pass "
|
|
146
|
-
"path::name to keep one." if label else "")
|
|
146
|
+
"path::name to keep one." if label else "")
|
|
147
|
+
+ (f" {code_graph.left_out_sentence(data)} Callers and tests beyond the cut are not listed." if data["truncated"] else ""),
|
|
148
|
+
**({"left_out": data["left_out"]} if data["truncated"] else {}),
|
|
147
149
|
}
|
|
148
150
|
|
|
149
151
|
|
package/disk_cleanup.py
CHANGED
|
@@ -1,12 +1,16 @@
|
|
|
1
1
|
"""Removes what Smart Tool left on disk that no registered project needs.
|
|
2
2
|
|
|
3
3
|
Runs once when the daemon starts, before the project monitor: the registry of projects (a folder that is not a
|
|
4
|
-
project, see project_admission.junk_reason, is never one of them, and each removal is listed in a
|
|
5
|
-
|
|
6
|
-
|
|
4
|
+
project or no longer exists, see project_admission.junk_reason, is never one of them, and each removal is listed in a
|
|
5
|
+
manifest with what re-registering the project needs), write temporaries orphaned by a crash, generations under
|
|
6
|
+
construction that stopped advancing, the `.previous` copies older versions kept of a replaced generation, index files
|
|
7
|
+
of projects no longer registered, and saved code-graph analyses of index views that no longer exist. Source files are
|
|
8
|
+
never touched."""
|
|
9
|
+
import datetime
|
|
7
10
|
import json
|
|
8
11
|
import os
|
|
9
12
|
import re
|
|
13
|
+
import sqlite3
|
|
10
14
|
import time
|
|
11
15
|
|
|
12
16
|
import atomic_io
|
|
@@ -26,6 +30,10 @@ _STATE_FILE = re.compile(r"([0-9a-f]{16})(?:\.v-[0-9a-f]{16})?(?:\.previous|\.bu
|
|
|
26
30
|
_BUILDING = re.compile(r"[0-9a-f]{16}(?:\.v-[0-9a-f]{16})?\.building-[0-9a-f]+\.sqlite3")
|
|
27
31
|
_PREVIOUS = re.compile(r"[0-9a-f]{16}(?:\.v-[0-9a-f]{16})?\.previous\.sqlite3")
|
|
28
32
|
_TEMPORARY = re.compile(r".+" + re.escape(atomic_io.TMP_SUFFIX) + r"[0-9a-f]{32}")
|
|
33
|
+
_VIEW_INDEX = re.compile(r"([0-9a-f]{16})(?:\.v-[0-9a-f]{16})?\.sqlite3")
|
|
34
|
+
_SCOPE_FILE = re.compile(r"([0-9a-f]{16})(?:\.v-[0-9a-f]{16})?\.scope\.json")
|
|
35
|
+
REGISTRATION_KEYS = ("watch", "paused", "enabled", "update_mode", "created_at", "last_indexed")
|
|
36
|
+
SCOPE_KEYS = ("include", "exclude", "user_exclude", "manual")
|
|
29
37
|
|
|
30
38
|
|
|
31
39
|
def _remove(path, report):
|
|
@@ -36,11 +44,15 @@ def _remove(path, report):
|
|
|
36
44
|
report["bytes"] += size
|
|
37
45
|
|
|
38
46
|
|
|
47
|
+
def _project_keys(root):
|
|
48
|
+
return {project_identity.project_id(root), *project_identity.legacy_ids(root)}
|
|
49
|
+
|
|
50
|
+
|
|
39
51
|
def remove_project_files(root, report=None):
|
|
40
52
|
"""Deletes every index, vector cache, scope and saved graph of the project at root (all Git views, the current and
|
|
41
53
|
legacy ids); returns the report {files, bytes}. Raises RuntimeError when another process holds one open."""
|
|
42
54
|
report = report if report is not None else {"files": [], "bytes": 0}
|
|
43
|
-
keys =
|
|
55
|
+
keys = _project_keys(root)
|
|
44
56
|
for name in sorted(os.listdir(indexer.INDEX_DIR)):
|
|
45
57
|
match = _STATE_FILE.fullmatch(name)
|
|
46
58
|
if not match or match.group(1) not in keys:
|
|
@@ -74,12 +86,50 @@ def junk_projects(errors=None):
|
|
|
74
86
|
return found
|
|
75
87
|
|
|
76
88
|
|
|
89
|
+
def recovery_record(entry):
|
|
90
|
+
"""What it takes to register the junk project again, read before its index files go: the junk entry with its
|
|
91
|
+
registration settings, the files and chunks its indexes hold (summed over Git views, `views` counts them) and the
|
|
92
|
+
scope of each view. An index that cannot be read is listed in `count_errors`, never counted as empty."""
|
|
93
|
+
project = project_store.get(entry["id"]) or {}
|
|
94
|
+
keys = _project_keys(entry["root"])
|
|
95
|
+
record = {**entry, "settings": {key: project[key] for key in REGISTRATION_KEYS if key in project},
|
|
96
|
+
"indexed_files": 0, "chunks": 0, "views": 0, "scopes": {}, "count_errors": []}
|
|
97
|
+
for name in sorted(os.listdir(indexer.INDEX_DIR)):
|
|
98
|
+
path = os.path.join(indexer.INDEX_DIR, name)
|
|
99
|
+
if (match := _VIEW_INDEX.fullmatch(name)) and match.group(1) in keys:
|
|
100
|
+
try:
|
|
101
|
+
conn = indexer._readonly(path)
|
|
102
|
+
try:
|
|
103
|
+
record["indexed_files"] += conn.execute("SELECT count(*) FROM manifest").fetchone()[0]
|
|
104
|
+
record["chunks"] += conn.execute("SELECT count(*) FROM chunks").fetchone()[0]
|
|
105
|
+
record["views"] += 1
|
|
106
|
+
finally:
|
|
107
|
+
conn.close()
|
|
108
|
+
except (OSError, sqlite3.Error) as exc:
|
|
109
|
+
record["count_errors"].append(f"{name}: {exc}")
|
|
110
|
+
elif (match := _SCOPE_FILE.fullmatch(name)) and match.group(1) in keys:
|
|
111
|
+
try:
|
|
112
|
+
with open(path, encoding="utf-8") as stream:
|
|
113
|
+
scope = json.load(stream)
|
|
114
|
+
record["scopes"][name] = {key: scope[key] for key in SCOPE_KEYS if key in scope}
|
|
115
|
+
except (OSError, ValueError) as exc:
|
|
116
|
+
record["count_errors"].append(f"{name}: {exc}")
|
|
117
|
+
return record
|
|
118
|
+
|
|
119
|
+
|
|
77
120
|
def remove_junk(entries, log=None):
|
|
78
121
|
"""Unregisters the given junk projects and deletes their index files (never the project's own files), then writes
|
|
79
|
-
a manifest of what was removed
|
|
80
|
-
|
|
122
|
+
a manifest of what was removed (the `recovery_record` of each, with the index files, bytes freed and the time) and
|
|
123
|
+
what was kept; returns {removed, kept, files, bytes, manifest}. The manifest is first written with every project
|
|
124
|
+
as `pending`, before anything is deleted, so a crash half way still leaves what is needed to recover. A project
|
|
125
|
+
whose index another process holds open is kept and listed with the reason."""
|
|
81
126
|
log = log or (lambda message: None)
|
|
82
127
|
report = {"removed": [], "kept": [], "files": 0, "bytes": 0, "manifest": None}
|
|
128
|
+
records = {entry["id"]: recovery_record(entry) for entry in entries}
|
|
129
|
+
if records:
|
|
130
|
+
os.makedirs(MANIFEST_DIR, exist_ok=True)
|
|
131
|
+
report["manifest"] = os.path.join(MANIFEST_DIR, time.strftime("junk-%Y%m%d-%H%M%S.json"))
|
|
132
|
+
_write_manifest(report["manifest"], {"pending": list(records.values()), "removed": [], "kept": []})
|
|
83
133
|
for entry in entries:
|
|
84
134
|
files = {"files": [], "bytes": 0}
|
|
85
135
|
try:
|
|
@@ -89,18 +139,20 @@ def remove_junk(entries, log=None):
|
|
|
89
139
|
log(f"disk cleanup: {entry['root']} kept: {exc}")
|
|
90
140
|
continue
|
|
91
141
|
project_store.forget(entry["id"])
|
|
92
|
-
report["removed"].append({**entry, "index_files": files["files"], "bytes": files["bytes"]
|
|
142
|
+
report["removed"].append({**records[entry["id"]], "index_files": files["files"], "bytes": files["bytes"],
|
|
143
|
+
"removed_at": datetime.datetime.now().astimezone().isoformat(timespec="seconds")})
|
|
93
144
|
report["files"] += len(files["files"])
|
|
94
145
|
report["bytes"] += files["bytes"]
|
|
95
146
|
log(f"disk cleanup: unregistered {entry['root']}: {entry['reason']}")
|
|
96
|
-
if report["
|
|
97
|
-
|
|
98
|
-
report["manifest"] = os.path.join(MANIFEST_DIR, time.strftime("junk-%Y%m%d-%H%M%S.json"))
|
|
99
|
-
atomic_io.write_text(report["manifest"], json.dumps(
|
|
100
|
-
{"removed": report["removed"], "kept": report["kept"], "source_files_touched": False}, indent=1))
|
|
147
|
+
if report["manifest"]:
|
|
148
|
+
_write_manifest(report["manifest"], {"removed": report["removed"], "kept": report["kept"]})
|
|
101
149
|
return report
|
|
102
150
|
|
|
103
151
|
|
|
152
|
+
def _write_manifest(path, body):
|
|
153
|
+
atomic_io.write_text(path, json.dumps({**body, "source_files_touched": False}, indent=1))
|
|
154
|
+
|
|
155
|
+
|
|
104
156
|
def sweep(extra_roots, log=None):
|
|
105
157
|
"""Removes the waste described in the module docstring and returns {junk, files, bytes, errors}.
|
|
106
158
|
|
package/doc_scope.py
CHANGED
|
@@ -690,7 +690,7 @@ def generate(root, view_id=None, limit=30, model=None):
|
|
|
690
690
|
summary["failed_checks"] = [c for c in validation["checks"] if c["status"] == "fail"]
|
|
691
691
|
return {"validation": summary, "folder": folder, "written": len(files), "removed": [f"{DOCS_DIR}/{r}" for r in removed], **counts,
|
|
692
692
|
"prose_pending_total": len(pending), "prose_pending": pending[:limit], "prose_dropped": dropped[:limit],
|
|
693
|
-
"truncated": bool(model.graph.get("truncated")),
|
|
693
|
+
"truncated": bool(model.graph.get("truncated")), "left_out": model.graph["left_out"],
|
|
694
694
|
"extras": sorted(f"{DOCS_DIR}/{rel}" for rel in extras), "index_file": index_file if block is not None else None,
|
|
695
695
|
"guidance": (f"Validation FAILED: {summary['claims_failed']} claim(s) do not hold; read {validation['report']} "
|
|
696
696
|
"and fix the generator or the code before trusting these pages. " if summary["verdict"] == "fail"
|
package/duplicates.py
CHANGED
|
@@ -184,7 +184,7 @@ def _functions(root, include_tests):
|
|
|
184
184
|
if inventory.get("message"):
|
|
185
185
|
notes.append(inventory["message"])
|
|
186
186
|
if graph.get("truncated"):
|
|
187
|
-
notes.append(
|
|
187
|
+
notes.append(code_graph.left_out_sentence(graph))
|
|
188
188
|
if graph.get("retryable"):
|
|
189
189
|
notes.append("Partial analysis: " + "; ".join(d.get("reason", "") for d in graph.get("diagnostics", [])[:2]))
|
|
190
190
|
return functions, notes, view
|
package/index_scope.py
CHANGED
|
@@ -12,6 +12,7 @@ import hashlib
|
|
|
12
12
|
import json
|
|
13
13
|
import os
|
|
14
14
|
import paths
|
|
15
|
+
import re
|
|
15
16
|
import time
|
|
16
17
|
|
|
17
18
|
import atomic_io
|
|
@@ -125,6 +126,151 @@ def always_excluded(name):
|
|
|
125
126
|
folded = name.casefold()
|
|
126
127
|
return any(fnmatch.fnmatch(folded, pattern.casefold()) for pattern in _ALWAYS_EXCLUDE)
|
|
127
128
|
|
|
129
|
+
|
|
130
|
+
def venv_name(name):
|
|
131
|
+
"""Whether a folder name marks a Python virtual environment: it holds `venv` (venv, venv-ov, gate-venv, .devenv)."""
|
|
132
|
+
return "venv" in name.casefold()
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
def is_venv_folder(path):
|
|
136
|
+
"""Whether the folder at path is a Python virtual environment, by its name (`venv_name`) or because it holds
|
|
137
|
+
`pyvenv.cfg`, which every virtual environment writes whatever its folder is called."""
|
|
138
|
+
return venv_name(os.path.basename(path)) or os.path.isfile(os.path.join(path, "pyvenv.cfg"))
|
|
139
|
+
|
|
140
|
+
|
|
141
|
+
def _is_exempt(rel_path, bases):
|
|
142
|
+
"""Whether rel_path is an include base or a folder above one: the scope names it, so no exclusion rule judges it."""
|
|
143
|
+
folded = rel_path.casefold()
|
|
144
|
+
return any(base.casefold() == folded or base.casefold().startswith(folded + "/") for base in bases if base)
|
|
145
|
+
|
|
146
|
+
|
|
147
|
+
def _inside_venv(root, parts, is_dir, bases):
|
|
148
|
+
"""Whether a folder on the way to the path (its last part counts only when it is a folder) is a virtual
|
|
149
|
+
environment, leaving out the folders `_is_exempt` spares."""
|
|
150
|
+
for depth in range(1, (len(parts) if is_dir else len(parts) - 1) + 1):
|
|
151
|
+
if not _is_exempt("/".join(parts[:depth]), bases) and is_venv_folder(os.path.join(root, *parts[:depth])):
|
|
152
|
+
return True
|
|
153
|
+
return False
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
def _gitignore_source(pattern):
|
|
157
|
+
"""Regular expression source of one `.gitignore` pattern, already stripped of `!` and the closing `/`, matched
|
|
158
|
+
against a path relative to the folder of the file: `*` and `?` stay inside a name, `**` crosses folders, `[...]`
|
|
159
|
+
is a class, `\\x` is a literal, and a pattern with a `/` is anchored to that folder."""
|
|
160
|
+
anchored = "/" in pattern
|
|
161
|
+
pattern = pattern.lstrip("/")
|
|
162
|
+
out, index = [], 0
|
|
163
|
+
while index < len(pattern):
|
|
164
|
+
char = pattern[index]
|
|
165
|
+
if pattern.startswith("**/", index):
|
|
166
|
+
out.append("(?:.*/)?")
|
|
167
|
+
index += 3
|
|
168
|
+
continue
|
|
169
|
+
if pattern.startswith("**", index):
|
|
170
|
+
out.append(".*")
|
|
171
|
+
index += 2
|
|
172
|
+
continue
|
|
173
|
+
close = pattern.find("]", index + 2) if char == "[" else -1
|
|
174
|
+
if char == "*":
|
|
175
|
+
out.append("[^/]*")
|
|
176
|
+
elif char == "?":
|
|
177
|
+
out.append("[^/]")
|
|
178
|
+
elif close != -1:
|
|
179
|
+
body = pattern[index + 1:close]
|
|
180
|
+
out.append("[" + ("^" + body[1:] if body.startswith("!") else body).replace("\\", "\\\\") + "]")
|
|
181
|
+
index = close
|
|
182
|
+
elif char == "\\" and index + 1 < len(pattern):
|
|
183
|
+
index += 1
|
|
184
|
+
out.append(re.escape(pattern[index]))
|
|
185
|
+
else:
|
|
186
|
+
out.append(re.escape(char))
|
|
187
|
+
index += 1
|
|
188
|
+
return ("" if anchored else "(?:.*/)?") + "".join(out)
|
|
189
|
+
|
|
190
|
+
|
|
191
|
+
def _gitignore_runs(text):
|
|
192
|
+
"""The rules of one `.gitignore` as runs of consecutive rules with the same polarity, in file order:
|
|
193
|
+
`[(negated, regex_for_any_entry, regex_for_files_or_None)]`. A pattern that is not a valid glob matches nothing,
|
|
194
|
+
as in Git."""
|
|
195
|
+
flags = re.IGNORECASE if os.name == "nt" else 0
|
|
196
|
+
runs = []
|
|
197
|
+
for raw in text.splitlines():
|
|
198
|
+
line = raw if raw.endswith("\\ ") else raw.rstrip(" \t")
|
|
199
|
+
if not line or line.startswith("#"):
|
|
200
|
+
continue
|
|
201
|
+
negated = line.startswith("!")
|
|
202
|
+
line = line[1:] if negated else line
|
|
203
|
+
dir_only = line.endswith("/")
|
|
204
|
+
line = line.rstrip("/")
|
|
205
|
+
if not line:
|
|
206
|
+
continue
|
|
207
|
+
source = _gitignore_source(line)
|
|
208
|
+
try:
|
|
209
|
+
re.compile(source, flags)
|
|
210
|
+
except re.error:
|
|
211
|
+
continue
|
|
212
|
+
if not runs or runs[-1][0] != negated:
|
|
213
|
+
runs.append((negated, [], []))
|
|
214
|
+
runs[-1][1].append(source)
|
|
215
|
+
if not dir_only:
|
|
216
|
+
runs[-1][2].append(source)
|
|
217
|
+
|
|
218
|
+
def combine(sources):
|
|
219
|
+
return re.compile("(?:" + "|".join(f"(?:{source})" for source in sources) + r")\Z", flags) if sources else None
|
|
220
|
+
|
|
221
|
+
return [(negated, combine(anywhere), combine(files)) for negated, anywhere, files in runs]
|
|
222
|
+
|
|
223
|
+
|
|
224
|
+
class GitIgnore:
|
|
225
|
+
"""What the `.gitignore` files inside a project folder ignore: the file of each folder applies to that folder and
|
|
226
|
+
below, a deeper file overrides a shallower one, and within a file the last matching rule wins. Read from the
|
|
227
|
+
project folder only (not Git's global excludes, `.git/info/exclude` or the files of a repository above the
|
|
228
|
+
project), a file is read when first needed and again after `TTL_S`, so one instance can serve a burst of file
|
|
229
|
+
events without going stale."""
|
|
230
|
+
|
|
231
|
+
TTL_S = 5
|
|
232
|
+
|
|
233
|
+
def __init__(self, root):
|
|
234
|
+
self.root = root
|
|
235
|
+
self._folders = {}
|
|
236
|
+
|
|
237
|
+
def _runs(self, folder):
|
|
238
|
+
now = time.monotonic()
|
|
239
|
+
hit = self._folders.get(folder)
|
|
240
|
+
if hit and now - hit[0] < self.TTL_S:
|
|
241
|
+
return hit[1]
|
|
242
|
+
try:
|
|
243
|
+
with open(os.path.join(self.root, folder, ".gitignore"), encoding="utf-8-sig", errors="replace") as stream:
|
|
244
|
+
text = stream.read()
|
|
245
|
+
except (FileNotFoundError, NotADirectoryError):
|
|
246
|
+
text = ""
|
|
247
|
+
runs = _gitignore_runs(text)
|
|
248
|
+
self._folders[folder] = (now, runs)
|
|
249
|
+
return runs
|
|
250
|
+
|
|
251
|
+
def matches(self, rel_path, is_dir):
|
|
252
|
+
"""Whether the rules ignore this entry by themselves, which is the question during a walk that never enters an
|
|
253
|
+
ignored folder. `rel_path` uses `/`."""
|
|
254
|
+
parts = rel_path.split("/")
|
|
255
|
+
verdict = False
|
|
256
|
+
for depth in range(len(parts)):
|
|
257
|
+
target = "/".join(parts[depth:])
|
|
258
|
+
for negated, anywhere, files in self._runs("/".join(parts[:depth])):
|
|
259
|
+
pattern = anywhere if is_dir else files
|
|
260
|
+
if pattern and pattern.match(target):
|
|
261
|
+
verdict = not negated
|
|
262
|
+
return verdict
|
|
263
|
+
|
|
264
|
+
def ignored(self, rel_path, is_dir, bases=()):
|
|
265
|
+
"""Whether the entry or any folder above it is ignored, for a path met out of a walk (a file event). The
|
|
266
|
+
include `bases` and the folders above them are spared: the scope names them."""
|
|
267
|
+
parts = rel_path.split("/")
|
|
268
|
+
for depth in range(1, len(parts) + 1):
|
|
269
|
+
sub = "/".join(parts[:depth])
|
|
270
|
+
if not _is_exempt(sub, bases) and self.matches(sub, depth < len(parts) or is_dir):
|
|
271
|
+
return True
|
|
272
|
+
return False
|
|
273
|
+
|
|
128
274
|
# Versão do contrato de decisão de escopo. Mudar o que é perguntado ao decisor exige
|
|
129
275
|
# incrementar: sem isso, um repositório com escopo já cacheado nunca é reavaliado, porque
|
|
130
276
|
# `needs_rescan` só olha diretório de topo novo.
|
|
@@ -194,6 +340,7 @@ def _top_level_dirs(root):
|
|
|
194
340
|
and not always_excluded(name)
|
|
195
341
|
and name not in skipped
|
|
196
342
|
and os.path.isdir(os.path.join(root, name))
|
|
343
|
+
and not is_venv_folder(os.path.join(root, name))
|
|
197
344
|
and _within_root(root, os.path.join(root, name))
|
|
198
345
|
)
|
|
199
346
|
|
|
@@ -259,6 +406,7 @@ def _structure(root, max_dirs=1500, keep=None):
|
|
|
259
406
|
found = []
|
|
260
407
|
for current, dirs, _files in os.walk(root):
|
|
261
408
|
dirs[:] = sorted(d for d in dirs if not d.startswith(".") and not always_excluded(d)
|
|
409
|
+
and not is_venv_folder(os.path.join(current, d))
|
|
262
410
|
and _within_root(root, os.path.join(current, d))
|
|
263
411
|
and (keep is None or keep(os.path.relpath(os.path.join(current, d), root).replace(os.sep, "/"))))
|
|
264
412
|
for name in dirs:
|
|
@@ -324,6 +472,7 @@ def _inspect_directory(root, rel_path, max_files=4000, sample_size=15, top_ext=1
|
|
|
324
472
|
for dirpath, _dirnames, filenames in os.walk(base_full):
|
|
325
473
|
_dirnames[:] = [name for name in _dirnames if not name.startswith(".")
|
|
326
474
|
and not always_excluded(name)
|
|
475
|
+
and not is_venv_folder(os.path.join(dirpath, name))
|
|
327
476
|
and _within_root(root, os.path.join(dirpath, name))]
|
|
328
477
|
depth = os.path.relpath(dirpath, base_full).count(os.sep) + (0 if dirpath == base_full else 1)
|
|
329
478
|
max_depth = max(max_depth, depth)
|
|
@@ -387,8 +536,10 @@ def _build_scope_prompt(root, ambiguous_dirs, root_files, doc_context):
|
|
|
387
536
|
"documentation.\n\n"
|
|
388
537
|
"`include` lists folder paths relative to the root, without wildcards; a folder you leave out of `include` is "
|
|
389
538
|
"not indexed, and an empty `include` indexes only the loose root files. "
|
|
390
|
-
"
|
|
391
|
-
"
|
|
539
|
+
"Whatever .gitignore ignores, and every folder named like *venv* or holding pyvenv.cfg (a Python virtual "
|
|
540
|
+
"environment), is not indexed, unless `include` names that folder or file: name the gitignored ones that are "
|
|
541
|
+
"useful for search (e.g. docs/, ADRs, CLAUDE.md). "
|
|
542
|
+
"Java, Angular, HTML/CSS, plain text and Markdown are supported sources. "
|
|
392
543
|
"DOCX, PPTX, XLSX and PDF documents are supported through bounded local text extraction; do not exclude "
|
|
393
544
|
"useful Office or PDF documentation merely because its container is binary. "
|
|
394
545
|
"Exclude directories that inspection shows are generated artifacts or "
|
|
@@ -555,15 +706,19 @@ def scope_filter(scope, root=None):
|
|
|
555
706
|
set(mechanical_excludes(root) if root else ()) | _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
|
|
556
707
|
whole = "" in bases
|
|
557
708
|
nested = _nested_matcher(root)
|
|
709
|
+
ignore = GitIgnore(root) if root else None
|
|
558
710
|
|
|
559
711
|
def keep(rel_path):
|
|
560
712
|
rel_path = rel_path.replace("\\", "/")
|
|
561
713
|
parts = rel_path.split("/")
|
|
562
714
|
if matcher(rel_path) or is_hidden(rel_path) or nested(rel_path):
|
|
563
715
|
return False
|
|
564
|
-
if whole or len(parts) == 1:
|
|
716
|
+
if not (whole or len(parts) == 1 or any(rel_path == base or rel_path.startswith(base + "/") for base in bases)):
|
|
717
|
+
return False
|
|
718
|
+
if not root:
|
|
565
719
|
return True
|
|
566
|
-
|
|
720
|
+
is_dir = os.path.isdir(os.path.join(root, *parts))
|
|
721
|
+
return not (_inside_venv(root, parts, is_dir, bases) or ignore.ignored(rel_path, is_dir, bases))
|
|
567
722
|
|
|
568
723
|
return keep
|
|
569
724
|
|
|
@@ -581,6 +736,7 @@ def resolve_included_files(root, scope, cancel_check=None):
|
|
|
581
736
|
excluded = _exclude_matcher(set(scope.get("exclude") or []) | set(scope.get("user_exclude") or []) |
|
|
582
737
|
set(mechanical_excludes(root)) | _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
|
|
583
738
|
nested = _nested_matcher(root)
|
|
739
|
+
ignore = GitIgnore(root)
|
|
584
740
|
|
|
585
741
|
def matcher(rel_path):
|
|
586
742
|
return excluded(rel_path) or nested(rel_path)
|
|
@@ -589,7 +745,7 @@ def resolve_included_files(root, scope, cancel_check=None):
|
|
|
589
745
|
def walk_error(error):
|
|
590
746
|
raise OSError(f"Could not list {error.filename} ({error.strerror}); try again when it is accessible.") from error
|
|
591
747
|
if "" not in include:
|
|
592
|
-
files.extend(_root_level_files(root, matcher))
|
|
748
|
+
files.extend(_root_level_files(root, lambda rel: matcher(rel) or ignore.matches(rel, False)))
|
|
593
749
|
for base in include:
|
|
594
750
|
base_full = os.path.join(root, base) if base else root
|
|
595
751
|
# `include` vem de tool call de um LLM cujo prompt embute documentação do
|
|
@@ -613,12 +769,15 @@ def resolve_included_files(root, scope, cancel_check=None):
|
|
|
613
769
|
d for d in dirnames
|
|
614
770
|
if not d.startswith(".")
|
|
615
771
|
and not matcher(f"{rel_dir}/{d}" if rel_dir else d)
|
|
772
|
+
and not is_venv_folder(os.path.join(dirpath, d))
|
|
773
|
+
and not ignore.matches(f"{rel_dir}/{d}" if rel_dir else d, True)
|
|
616
774
|
# `os.walk` não desce em symlink de diretório, mas `os.path.islink()` é
|
|
617
775
|
# False para junction do Windows (`mklink /J`, sem admin) — e desce nela.
|
|
618
776
|
and _within_root(root, os.path.join(dirpath, d))
|
|
619
777
|
]
|
|
620
778
|
for name in filenames:
|
|
621
779
|
rel_path = f"{rel_dir}/{name}" if rel_dir else name
|
|
622
|
-
if not name.startswith(".") and not matcher(rel_path) and
|
|
780
|
+
if (not name.startswith(".") and not matcher(rel_path) and not ignore.matches(rel_path, False)
|
|
781
|
+
and _within_root(root, os.path.join(root, rel_path))):
|
|
623
782
|
files.append(rel_path)
|
|
624
783
|
return sorted(set(files))
|
package/layers.py
CHANGED
|
@@ -165,7 +165,7 @@ def summary(rule, graph, read):
|
|
|
165
165
|
the declared order, 'up' goes against it) and the folders no layer covers.
|
|
166
166
|
|
|
167
167
|
Parameters: rule (Rule); graph (dict) code graph with 'symbols' and 'calls'; read (callable path -> str or None).
|
|
168
|
-
Returns: dict with 'layers', 'calls_between', 'unassigned' and '
|
|
168
|
+
Returns: dict with 'layers', 'calls_between', 'unassigned', 'truncated' and 'left_out' (what the graph limits cut).
|
|
169
169
|
"""
|
|
170
170
|
order = {layer["name"]: index for index, layer in enumerate(rule.layers)}
|
|
171
171
|
paths = sorted({s["path"] for s in graph.get("symbols") or []})
|
|
@@ -185,7 +185,7 @@ def summary(rule, graph, read):
|
|
|
185
185
|
for (a, b), n in sorted(between.items(), key=lambda item: (order[item[0][0]],
|
|
186
186
|
order[item[0][1]]))],
|
|
187
187
|
"unassigned": [{"folder": folder, "files": n} for folder, n in unassigned.most_common(MAX_UNASSIGNED)],
|
|
188
|
-
"unassigned_files": files[None], "truncated": bool(graph.get("truncated"))}
|
|
188
|
+
"unassigned_files": files[None], "truncated": bool(graph.get("truncated")), "left_out": graph["left_out"]}
|
|
189
189
|
|
|
190
190
|
|
|
191
191
|
def report(root, layers=None, confirm=False, view_id=None):
|
|
@@ -229,7 +229,7 @@ def report(root, layers=None, confirm=False, view_id=None):
|
|
|
229
229
|
proposed = suggest(sorted({s["path"] for s in symbols}), read)
|
|
230
230
|
if not proposed:
|
|
231
231
|
return {"saved": False, "rule_path": path, "layers": [], "calls_between": [], "unassigned": [],
|
|
232
|
-
"truncated": bool(graph.get("truncated")),
|
|
232
|
+
"truncated": bool(graph.get("truncated")), "left_out": graph["left_out"],
|
|
233
233
|
"guidance": "No folder or annotation follows a known layer convention; declare the layers yourself "
|
|
234
234
|
"(name, paths, annotations, highest first) and confirm them."}
|
|
235
235
|
return {"saved": False, "rule_path": path, **summary(Rule(proposed), code, read),
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@allansantos-dev/smart-tool",
|
|
3
|
-
"version": "0.9.
|
|
3
|
+
"version": "0.9.79",
|
|
4
4
|
"description": "Local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharper tools than their built-in search. Windows.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"author": "Smart Tool contributors",
|
package/pr_impact.py
CHANGED
|
@@ -78,6 +78,8 @@ def report(root, base="HEAD", limit=30, view_id=None):
|
|
|
78
78
|
lines.append("No test reaches the changed code through imports or static calls.")
|
|
79
79
|
if changes["run_all"]:
|
|
80
80
|
lines += ["", "Run the whole suite: " + "; ".join(changes["run_all_reasons"]) + "."]
|
|
81
|
+
if changes.get("truncated"):
|
|
82
|
+
lines += ["", changes["note"]]
|
|
81
83
|
return {"base": base, "markdown": "\n".join(lines).rstrip("\n") + "\n", "touched": touched,
|
|
82
84
|
"callers": {label: (reached[1] if reached else 0) for label, reached in callers.items()},
|
|
83
85
|
"stale_anchors": stale, "stale_note": stale_note, "tests": changes["tests"],
|
package/project_admission.py
CHANGED
|
@@ -3,14 +3,17 @@
|
|
|
3
3
|
A project is the folder where the agent session started (or the repository that holds it), never a folder an agent
|
|
4
4
|
merely navigated to: personal folders, drive roots, application data, system folders, build output and a Java home
|
|
5
5
|
are refused, a subfolder resolves to the registered project or the repository that holds it, and a session whose
|
|
6
|
-
origin is known only indexes its own project. Registered folders that break these rules
|
|
6
|
+
origin is known only indexes its own project. Registered folders that break these rules, and registered folders that no
|
|
7
|
+
longer exist, are junk (`junk_reason`)."""
|
|
7
8
|
import os
|
|
8
9
|
import tempfile
|
|
9
10
|
|
|
10
11
|
import project_identity
|
|
11
12
|
import project_store
|
|
13
|
+
import redirect_rule
|
|
12
14
|
|
|
13
15
|
PERSONAL_NAMES = ("Desktop", "Documents", "Downloads", "Pictures", "Videos", "Music")
|
|
16
|
+
GONE_REASON = "its folder no longer exists"
|
|
14
17
|
TOOL_FOLDERS = frozenset(("bin", "obj", "node_modules", ".git", "site-packages", "__pycache__"))
|
|
15
18
|
POSIX_SYSTEM = ("/usr", "/bin", "/sbin", "/etc", "/var", "/opt", "/lib", "/proc", "/sys", "/dev", "/boot")
|
|
16
19
|
|
|
@@ -100,6 +103,17 @@ def refusal(root):
|
|
|
100
103
|
return None
|
|
101
104
|
|
|
102
105
|
|
|
106
|
+
def temporary_refusal(root):
|
|
107
|
+
"""Why a new project cannot live in root, or None: a copy made in the system's temporary folder for a measurement
|
|
108
|
+
or a build is gone soon and would leave its index behind. Folders of `SMART_TOOL_SCRATCH` stay allowed, and a
|
|
109
|
+
registered project is not asked (see `admit`)."""
|
|
110
|
+
norm = _norm(project_identity.display_root(root))
|
|
111
|
+
if _inside(norm, _norm(project_identity.display_root(redirect_rule.system_temp()))) and not any(
|
|
112
|
+
_inside(norm, base) for base in _env_folders("SMART_TOOL_SCRATCH")):
|
|
113
|
+
return "inside the system temporary folder"
|
|
114
|
+
return None
|
|
115
|
+
|
|
116
|
+
|
|
103
117
|
def repository_root(path):
|
|
104
118
|
"""The nearest folder at or above path that holds `.git` and could be a project, or None. A repository the rules
|
|
105
119
|
refuse (a dotfiles repository in the home folder) is not a project to resolve to."""
|
|
@@ -134,7 +148,7 @@ def admit(path, origin=None):
|
|
|
134
148
|
root, registered = resolve(path)
|
|
135
149
|
if registered:
|
|
136
150
|
return root
|
|
137
|
-
reason = refusal(root)
|
|
151
|
+
reason = refusal(root) or temporary_refusal(root)
|
|
138
152
|
if reason:
|
|
139
153
|
raise ValueError(f"Not indexing {root}: it is {reason}. A project is the folder where the session started; "
|
|
140
154
|
"pass that folder as project_root.")
|
|
@@ -151,10 +165,29 @@ def admit(path, origin=None):
|
|
|
151
165
|
return root
|
|
152
166
|
|
|
153
167
|
|
|
168
|
+
def folder_gone(root):
|
|
169
|
+
"""Whether root is confirmed missing, not merely unreachable. Both must hold: looking at root fails with "does not
|
|
170
|
+
exist", and the drive or network share root it sits on (`D:\\`, `\\\\server\\share\\`, `/`) is present, so the
|
|
171
|
+
missing part is below a mounted volume. A drive that is not mounted, a share that is offline and any other error
|
|
172
|
+
(access denied, device not ready) are not "gone": the folder may come back."""
|
|
173
|
+
try:
|
|
174
|
+
os.stat(root)
|
|
175
|
+
return False
|
|
176
|
+
except (FileNotFoundError, NotADirectoryError):
|
|
177
|
+
pass
|
|
178
|
+
except OSError:
|
|
179
|
+
return False
|
|
180
|
+
drive = os.path.splitdrive(os.path.abspath(root))[0]
|
|
181
|
+
return os.path.isdir(drive + os.sep)
|
|
182
|
+
|
|
183
|
+
|
|
154
184
|
def junk_reason(project):
|
|
155
|
-
"""Why a registered project was never a project, or None: its folder
|
|
156
|
-
more projects, or it is a subfolder of a repository or of another
|
|
185
|
+
"""Why a registered project was never a project, or None: its folder no longer exists (`folder_gone`), its folder
|
|
186
|
+
is refused by `refusal`, it holds two or more projects, or it is a subfolder of a repository or of another
|
|
187
|
+
registered project. A project whose folder exists is judged by the last three rules only."""
|
|
157
188
|
root = project["root"]
|
|
189
|
+
if folder_gone(root):
|
|
190
|
+
return GONE_REASON
|
|
158
191
|
if not os.path.isdir(root):
|
|
159
192
|
return None
|
|
160
193
|
reason = refusal(root)
|
package/project_monitor.py
CHANGED
|
@@ -35,6 +35,7 @@ def relevant_path(path):
|
|
|
35
35
|
parts = path.replace("\\", "/").strip("/").split("/")
|
|
36
36
|
return bool(parts and all(part not in ("", ".", "..") and not part.startswith(".")
|
|
37
37
|
and not index_scope.always_excluded(part) for part in parts)
|
|
38
|
+
and not any(index_scope.venv_name(part) for part in parts[:-1])
|
|
38
39
|
and not index_scope.is_transient(parts[-1]))
|
|
39
40
|
|
|
40
41
|
|
package/smart_tool_daemon.py
CHANGED
|
@@ -373,7 +373,7 @@ SERVER_INSTRUCTIONS = (
|
|
|
373
373
|
TOOLS = [
|
|
374
374
|
{
|
|
375
375
|
"name": "project_manage",
|
|
376
|
-
"description": "Manages folders and indexing. Every action takes project_root (the folder) or project_id. list gives a short line per project (a project whose folder is gone shows as unavailable with the reason); status shows storage, scope and jobs, and with a path inside a registered project answers for the deepest one that holds it (resolved_from). graph with symbol (function, Class.method, class name, or path::name when the name repeats) answers what an edit touches: definition, callers with line, calls, tests that reach it through static calls (depth hops, default 3) and files importing it; use it before changing a function to know what to update. affected_tests answers which tests to run after editing: every test file importing a changed file (git diff against base, default HEAD, untracked included) plus changed tests, likeliest failures first, with the command to run them and run_all when config or unanalyzed code changed. pr_impact (base, default HEAD; main or origin/main for a branch) returns a Markdown report of the change for a pull request: the functions the diff touches with their callers, the documentation entries it leaves out of date and the tests to run with their command; it publishes nothing. graph without symbol reads the indexed view: coverage, references and static calls in Java, Angular, Python, Go and JS/TS/JSX/TSX; includes HTML/CSS/Markdown references and TXT/DOCX coverage; with file_path, narrows to that file's symbols, imports and calls (lists capped by limit). inspect lists the files of a view_id; with file_path, returns only that file's chunks. usage shows consumption and reuse; docs lists public functions without a docstring (Python docstring, JSDoc, Javadoc, Go doc comment starting with the name; private, nested and override functions excluded) with coverage percent, excluding tests unless include_tests=true, for documenting a project that started without it; docs generate=true writes the documentation scope to docs/scope in the project in three levels tied to the code: technical pages by folder (an entry per public function anchored on path::name with signature, first doc line, layer, callers, calls and tests, plus a Mermaid flow by layer), functional.md (each entry point and the functions its flow reaches, by layer) and simple.md; the prose between the prose markers is kept by regeneration; docs write_prose=true writes the empty prose blocks with the Docs model of the setup (each paragraph checked against the code first) within about 40 s and says how many are left, call it again until none is; docs check=true reports anchors whose function changed signature or is gone and broken links; docs validate=true checks every claim of docs/scope against the code (anchors, links, line numbers, callers and tests, coverage, entry points, command-line options, prose names, counts, glossary, navigation, versions) and writes docs/scope/validation.md with pass/fail per check, the evidence, a score and the verdict (generate runs it too); docs with symbol lists where that function is documented; layers declares the project's layers (ordered highest first, each with path globs and annotations): without layers it returns the saved rule or a suggestion read from folder names and annotations, with files, symbols and calls between layers (direction down or up); pass layers to preview a rule and the same layers with confirm=true to save it in the project (.smart-tool/layers.json); duplicates lists duplicated functions (identical, near-identical and semantic bodies, min_similarity default 0.90; below that most pairs measured were false positives), excluding tests unless include_tests=true; fix real duplicates; only dismiss with duplicates_dismiss (finding_id, reason=false_positive|intentional, note) what is not a duplicate or is a copy kept on purpose: the finding stays hidden until the code changes, and duplicates_restore reopens it; integration shows the MCP client, hook and tool usage metrics; install_hook_preview shows the change that would install the hook redirecting Grep/Read to smart_search and WebSearch/WebFetch to web_search/web_fetch in this session's client and returns a plan_id; install_hook writes it with that plan_id and confirm=true, with a backup, only after the user approves the change; search_limits saves the project's default top_k per block ({code, test, doc}, each 1-20, total at most 20); storage shows disk usage and views; pin_view protects a view; each view carries stale (not updated for 30 days, the test of the daily cleanup that removes them) and stale_bytes sums them; cleanup_preview and cleanup_commit remove selected views after confirmation; compare compares two stored views, without checkout or embeddings. register adds a folder; preview checks the scope; index/rebuild update; pause/cancel keep checkpoints; resume continues; policy picks on_search/eager; scope adjusts the scope; remove deletes only the local index. setup_audit (project_root) reports what Claude Code and Codex actually load for that folder (settings layers, plugins, hooks, MCP servers, skills, CLAUDE.md/AGENTS.md) and where they collide: hooks deciding on the same tool in parallel, missing hook scripts, silenced hooks, MCP servers down or offering duplicate tools, skill name and budget problems, AGENTS.md that only Codex reads, broken @imports; what could not be checked is listed as not verified; each finding carries its fix (the file to edit or the command to run), applied by the agent when the user agrees, never by Smart Tool; mode=last returns the stored report instead of auditing again. redirect_proposals (no project_root) lists the redirect patterns proposed from allowed searches, most matched first, with a guide; redirect_decide decides them by ids: accept, instruct (tip without blocking), reject, or remove an accepted one. junk (no project_root) lists the registered projects that were never projects: a personal, system, application-data or build folder, a subfolder of a repository or of another project, a folder of projects; with confirm=true it unregisters them and deletes only their index files, and the manifest says what went. A project is the folder where the session started, so smart_search and register refuse a folder that is not a project and index a new one only from the session that started in it. Jobs via smart_search_result.",
|
|
376
|
+
"description": "Manages folders and indexing. Every action takes project_root (the folder) or project_id. list gives a short line per project (a project whose folder is gone shows as unavailable with the reason); status shows storage, scope and jobs, and with a path inside a registered project answers for the deepest one that holds it (resolved_from). graph with symbol (function, Class.method, class name, or path::name when the name repeats) answers what an edit touches: definition, callers with line, calls, tests that reach it through static calls (depth hops, default 3) and files importing it; use it before changing a function to know what to update. affected_tests answers which tests to run after editing: every test file importing a changed file (git diff against base, default HEAD, untracked included) plus changed tests, likeliest failures first, with the command to run them and run_all when config or unanalyzed code changed. pr_impact (base, default HEAD; main or origin/main for a branch) returns a Markdown report of the change for a pull request: the functions the diff touches with their callers, the documentation entries it leaves out of date and the tests to run with their command; it publishes nothing. graph without symbol reads the indexed view: coverage, references and static calls in Java, Angular, Python, Go and JS/TS/JSX/TSX; includes HTML/CSS/Markdown references and TXT/DOCX coverage; with file_path, narrows to that file's symbols, imports and calls (lists capped by limit). inspect lists the files of a view_id; with file_path, returns only that file's chunks. usage shows consumption and reuse; docs lists public functions without a docstring (Python docstring, JSDoc, Javadoc, Go doc comment starting with the name; private, nested and override functions excluded) with coverage percent, excluding tests unless include_tests=true, for documenting a project that started without it; docs generate=true writes the documentation scope to docs/scope in the project in three levels tied to the code: technical pages by folder (an entry per public function anchored on path::name with signature, first doc line, layer, callers, calls and tests, plus a Mermaid flow by layer), functional.md (each entry point and the functions its flow reaches, by layer) and simple.md; the prose between the prose markers is kept by regeneration; docs write_prose=true writes the empty prose blocks with the Docs model of the setup (each paragraph checked against the code first) within about 40 s and says how many are left, call it again until none is; docs check=true reports anchors whose function changed signature or is gone and broken links; docs validate=true checks every claim of docs/scope against the code (anchors, links, line numbers, callers and tests, coverage, entry points, command-line options, prose names, counts, glossary, navigation, versions) and writes docs/scope/validation.md with pass/fail per check, the evidence, a score and the verdict (generate runs it too); docs with symbol lists where that function is documented; layers declares the project's layers (ordered highest first, each with path globs and annotations): without layers it returns the saved rule or a suggestion read from folder names and annotations, with files, symbols and calls between layers (direction down or up); pass layers to preview a rule and the same layers with confirm=true to save it in the project (.smart-tool/layers.json); duplicates lists duplicated functions (identical, near-identical and semantic bodies, min_similarity default 0.90; below that most pairs measured were false positives), excluding tests unless include_tests=true; fix real duplicates; only dismiss with duplicates_dismiss (finding_id, reason=false_positive|intentional, note) what is not a duplicate or is a copy kept on purpose: the finding stays hidden until the code changes, and duplicates_restore reopens it; integration shows the MCP client, hook and tool usage metrics; install_hook_preview shows the change that would install the hook redirecting Grep/Read to smart_search and WebSearch/WebFetch to web_search/web_fetch in this session's client and returns a plan_id; install_hook writes it with that plan_id and confirm=true, with a backup, only after the user approves the change; search_limits saves the project's default top_k per block ({code, test, doc}, each 1-20, total at most 20); storage shows disk usage and views; pin_view protects a view; each view carries stale (not updated for 30 days, the test of the daily cleanup that removes them) and stale_bytes sums them; cleanup_preview and cleanup_commit remove selected views after confirmation; compare compares two stored views, without checkout or embeddings. register adds a folder; preview checks the scope; index/rebuild update; pause/cancel keep checkpoints; resume continues; policy picks on_search/eager; scope adjusts the scope; remove deletes only the local index. setup_audit (project_root) reports what Claude Code and Codex actually load for that folder (settings layers, plugins, hooks, MCP servers, skills, CLAUDE.md/AGENTS.md) and where they collide: hooks deciding on the same tool in parallel, missing hook scripts, silenced hooks, MCP servers down or offering duplicate tools, skill name and budget problems, AGENTS.md that only Codex reads, broken @imports; what could not be checked is listed as not verified; each finding carries its fix (the file to edit or the command to run), applied by the agent when the user agrees, never by Smart Tool; mode=last returns the stored report instead of auditing again. redirect_proposals (no project_root) lists the redirect patterns proposed from allowed searches, most matched first, with a guide; redirect_decide decides them by ids: accept, instruct (tip without blocking), reject, or remove an accepted one. junk (no project_root) lists the registered projects that were never projects: a personal, system, application-data or build folder, a subfolder of a repository or of another project, a folder of projects, or a folder that no longer exists (on a drive or share that is mounted); with confirm=true it unregisters them and deletes only their index files, and the manifest says what went. A project is the folder where the session started, so smart_search and register refuse a folder that is not a project and index a new one only from the session that started in it. Jobs via smart_search_result.",
|
|
377
377
|
"inputSchema": {
|
|
378
378
|
"type": "object", "properties": {
|
|
379
379
|
"action": {"type": "string", "enum": ["list", "status", "register", "preview", "index", "rebuild", "pause", "cancel", "resume", "watch", "scope", "remove", "relocate", "probe", "policy", "inspect", "graph", "usage", "storage", "pin_view", "cleanup_preview", "cleanup_commit", "compare", "search_limits", "integration", "install_hook_preview", "install_hook", "duplicates", "duplicates_dismiss", "duplicates_restore", "docs", "layers", "affected_tests", "pr_impact", "setup_audit", "redirect_proposals", "redirect_decide", "junk"]},
|
|
@@ -2191,9 +2191,9 @@ def _audited_roots():
|
|
|
2191
2191
|
|
|
2192
2192
|
def _junk_action(arguments):
|
|
2193
2193
|
"""`junk` lists the registered projects that were never projects (a personal or system folder, a subfolder of
|
|
2194
|
-
a repository or of another project, a folder of projects); with confirm=true it
|
|
2195
|
-
index files, never the project's own files, and returns the path of the
|
|
2196
|
-
kept and listed."""
|
|
2194
|
+
a repository or of another project, a folder of projects) or whose folder no longer exists; with confirm=true it
|
|
2195
|
+
unregisters them and deletes their index files, never the project's own files, and returns the path of the
|
|
2196
|
+
manifest, which records what re-registering each needs. A project with a running job is kept and listed."""
|
|
2197
2197
|
entries = disk_cleanup.junk_projects()
|
|
2198
2198
|
if arguments.get("confirm") is not True:
|
|
2199
2199
|
return {"junk": entries, "detail": "Nothing was removed. Pass confirm=true to unregister these and delete "
|
package/version.py
CHANGED