lscope 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- lscope-0.1.0/PKG-INFO +321 -0
- lscope-0.1.0/README.md +298 -0
- lscope-0.1.0/lscope.egg-info/PKG-INFO +321 -0
- lscope-0.1.0/lscope.egg-info/SOURCES.txt +10 -0
- lscope-0.1.0/lscope.egg-info/dependency_links.txt +1 -0
- lscope-0.1.0/lscope.egg-info/entry_points.txt +2 -0
- lscope-0.1.0/lscope.egg-info/requires.txt +16 -0
- lscope-0.1.0/lscope.egg-info/top_level.txt +1 -0
- lscope-0.1.0/main.py +1052 -0
- lscope-0.1.0/pyproject.toml +35 -0
- lscope-0.1.0/setup.cfg +4 -0
- lscope-0.1.0/test/test_main.py +68 -0
lscope-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,321 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: lscope
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: ast-grep using ladybug
|
|
5
|
+
Requires-Python: >=3.14
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: icebug>=12.9
|
|
8
|
+
Requires-Dist: ladybug>=0.17.1
|
|
9
|
+
Requires-Dist: polars>=1.41.2
|
|
10
|
+
Requires-Dist: pyarrow>=24.0.0
|
|
11
|
+
Requires-Dist: tqdm>=4.68.3
|
|
12
|
+
Requires-Dist: tree-sitter>=0.25.2
|
|
13
|
+
Requires-Dist: tree-sitter-c-sharp>=0.23.1
|
|
14
|
+
Requires-Dist: tree-sitter-cpp>=0.23.4
|
|
15
|
+
Requires-Dist: tree-sitter-go>=0.23.4
|
|
16
|
+
Requires-Dist: tree-sitter-java>=0.23.5
|
|
17
|
+
Requires-Dist: tree-sitter-javascript>=0.23.1
|
|
18
|
+
Requires-Dist: tree-sitter-kotlin>=1.1.0
|
|
19
|
+
Requires-Dist: tree-sitter-python>=0.25.0
|
|
20
|
+
Requires-Dist: tree-sitter-rust>=0.24.2
|
|
21
|
+
Requires-Dist: tree-sitter-swift>=0.0.1
|
|
22
|
+
Requires-Dist: tree-sitter-typescript>=0.23.2
|
|
23
|
+
|
|
24
|
+
# lscope — semantic code graph (via tree-sitter) on Ladybug
|
|
25
|
+
|
|
26
|
+
lscope parses source files with tree-sitter, extracts a semantic graph
|
|
27
|
+
(files, classes, functions, methods, calls), and stores it in a
|
|
28
|
+
[Ladybug](https://github.com/thatguyfrombb/ladybug) database. It then
|
|
29
|
+
supports lightweight code-intelligence queries: find functions by name
|
|
30
|
+
pattern, or find callers of a given function.
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Pipeline phases
|
|
35
|
+
|
|
36
|
+
When invoked with `--index`, the tool runs through several phases.
|
|
37
|
+
Some are parallel, some are single-threaded. The diagram below shows
|
|
38
|
+
the data flow and concurrency model:
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
42
|
+
│ 1. File Discovery (single-threaded) │
|
|
43
|
+
│ Walk targets, match extensions, collect file list │
|
|
44
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
45
|
+
│
|
|
46
|
+
▼
|
|
47
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
48
|
+
│ 2. Analysis / Parsing (parallel — ThreadPoolExecutor) │
|
|
49
|
+
│ Each file is read + tree-sitter parsed in a worker thread. │
|
|
50
|
+
│ Every thread has its own thread-local LanguageRegistry │
|
|
51
|
+
│ (parser instances). │
|
|
52
|
+
│ Progress: tqdm "Analyzing" bar across all files. │
|
|
53
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
54
|
+
│
|
|
55
|
+
▼
|
|
56
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
57
|
+
│ 3. Node Ingestion (parallel when workers > 1) │
|
|
58
|
+
│ Analyses are split round-robin into worker chunks. │
|
|
59
|
+
│ Each chunk runs in its own thread, opening its own │
|
|
60
|
+
│ ladybug Connection with enable_multi_writes=True. │
|
|
61
|
+
│ Nodes are bulk-inserted via UNWIND + MERGE per label │
|
|
62
|
+
│ in a single transaction per chunk. │
|
|
63
|
+
│ Progress: tqdm "Ingesting" bar across chunks. │
|
|
64
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
65
|
+
│
|
|
66
|
+
▼
|
|
67
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
68
|
+
│ 4. Definition Edges (single-threaded) │
|
|
69
|
+
│ CONTAINS / DEFINES / HAS_METHOD edges inserted via │
|
|
70
|
+
│ COPY CodeRelation (DDL — cannot run concurrently with │
|
|
71
|
+
│ active write transactions from step 3). │
|
|
72
|
+
│ Runs on the main connection after all workers finish. │
|
|
73
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
74
|
+
│
|
|
75
|
+
▼
|
|
76
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
77
|
+
│ 5. Call Resolution & Edges (single-threaded) │
|
|
78
|
+
│ Build a global name index across all analyzed files. │
|
|
79
|
+
│ Resolve each call expression to a declaration by name, │
|
|
80
|
+
│ preferring same-file matches. Bulk-insert CALLS edges │
|
|
81
|
+
│ via COPY CodeRelation on the main connection. │
|
|
82
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
When running a **query** (`--find-functions` or `--find-callers`), the
|
|
86
|
+
pipeline is trivial: a single read-only connection is opened and the
|
|
87
|
+
query executes on the main thread.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
### Phase details
|
|
92
|
+
|
|
93
|
+
#### 1. File discovery — single-threaded
|
|
94
|
+
|
|
95
|
+
`iter_source_files()` walks the given paths (files or directories).
|
|
96
|
+
Directories are recursed with `os.walk`, skipping common ignorable
|
|
97
|
+
directories (`.git`, `__pycache__`, `node_modules`, `.venv`, …).
|
|
98
|
+
Files are matched against the known extension map from the installed
|
|
99
|
+
tree-sitter grammars. If `--language` is given, only files of that
|
|
100
|
+
language are kept.
|
|
101
|
+
|
|
102
|
+
This runs entirely in the main thread — it is I/O-bound on `os.walk`
|
|
103
|
+
and has no heavy computation.
|
|
104
|
+
|
|
105
|
+
#### 2. Analysis / parsing — parallel (ThreadPoolExecutor)
|
|
106
|
+
|
|
107
|
+
Each source file is read from disk and parsed by a tree-sitter `Parser`.
|
|
108
|
+
The parser is obtained from a **thread-local** `LanguageRegistry`
|
|
109
|
+
(`_worker_state.registry`) so each thread creates its own parser
|
|
110
|
+
instances — tree-sitter parsers are not thread-safe for concurrent
|
|
111
|
+
use from multiple threads.
|
|
112
|
+
|
|
113
|
+
The `analyze_path()` function:
|
|
114
|
+
1. Determines the language from the file extension.
|
|
115
|
+
2. Reads the full file into a string.
|
|
116
|
+
3. Parses it with the appropriate tree-sitter grammar.
|
|
117
|
+
4. Walks the CST to extract:
|
|
118
|
+
- **Containers** — classes, structs, interfaces, traits, impl blocks.
|
|
119
|
+
- **Functions / methods** — with their owning container.
|
|
120
|
+
- **Call expressions** — recording caller, callee name, and source location.
|
|
121
|
+
|
|
122
|
+
No database calls happen in this phase. All extracted data is returned
|
|
123
|
+
as plain dicts for the next phase.
|
|
124
|
+
|
|
125
|
+
#### 3. Node ingestion — parallel (ThreadPoolExecutor, multi-write DB)
|
|
126
|
+
|
|
127
|
+
`ingest_analyses_parallel()` splits the list of analyses into
|
|
128
|
+
`workers` chunks (round-robin). Each chunk is processed by a worker
|
|
129
|
+
thread that:
|
|
130
|
+
|
|
131
|
+
1. Opens its own `ladybug.Connection` to the shared database (which was
|
|
132
|
+
opened with `enable_multi_writes=True`).
|
|
133
|
+
2. Calls `_collect_file_data()` to organise extracted symbols by label.
|
|
134
|
+
3. Bulk-inserts all nodes for that chunk:
|
|
135
|
+
- **Files** — `UNWIND … MERGE (f:File …)`.
|
|
136
|
+
- **Symbols** — `UNWIND … MERGE (n:{label} …)` per label
|
|
137
|
+
(Function, Method, Class, Struct, …).
|
|
138
|
+
- All wrapped in a single `BEGIN TRANSACTION … COMMIT` per chunk.
|
|
139
|
+
4. Returns definition edges for deferred insertion.
|
|
140
|
+
|
|
141
|
+
**Why parallel?** Node insertion is the bulk of the write work and
|
|
142
|
+
Ladybug's multi-write mode permits concurrent write transactions from
|
|
143
|
+
different connections. This gives a significant speedup for large
|
|
144
|
+
codebases.
|
|
145
|
+
|
|
146
|
+
**Why not parallel for everything?** Step 4 (definition edges) and
|
|
147
|
+
step 5 (call edges) use `COPY CodeRelation`, which is a DDL statement
|
|
148
|
+
and cannot run concurrently with other write transactions.
|
|
149
|
+
|
|
150
|
+
#### 4. Definition edges — single-threaded
|
|
151
|
+
|
|
152
|
+
After all worker threads finish, the main thread inserts edges that
|
|
153
|
+
represent:
|
|
154
|
+
|
|
155
|
+
- **CONTAINS** — file contains a function/class/etc.
|
|
156
|
+
- **DEFINES** — a file defines a top-level symbol.
|
|
157
|
+
- **HAS_METHOD** — a container (class/struct/trait/impl) has a method.
|
|
158
|
+
|
|
159
|
+
Edges are grouped by `(source_label, target_label)`, written to a
|
|
160
|
+
temporary Parquet file per group, and bulk-loaded with
|
|
161
|
+
`COPY CodeRelation FROM '…' (FROM='…', TO='…')`.
|
|
162
|
+
|
|
163
|
+
This runs single-threaded because `COPY` on the `CodeRelation`
|
|
164
|
+
relationship table group is a DDL-level operation that requires
|
|
165
|
+
exclusive write access.
|
|
166
|
+
|
|
167
|
+
#### 5. Call resolution & edges — single-threaded
|
|
168
|
+
|
|
169
|
+
`ingest_calls()` builds a global dictionary mapping each declared
|
|
170
|
+
function/method name to its symbol dict(s), then iterates over every
|
|
171
|
+
call expression from every file:
|
|
172
|
+
|
|
173
|
+
1. Looks up the callee name in the name index.
|
|
174
|
+
2. Prefers a declaration in the **same file** (for local disambiguation).
|
|
175
|
+
3. Falls back to the first candidate if there is no same-file match.
|
|
176
|
+
4. Assigns `confidence = 1.0` for unique matches, `0.7` when multiple
|
|
177
|
+
candidates exist (ambiguous name).
|
|
178
|
+
5. Bulk-inserts all `CALLS` edges via `_ingest_chunk_edges()` — again
|
|
179
|
+
using the DDL `COPY` path.
|
|
180
|
+
|
|
181
|
+
This must be single-threaded because:
|
|
182
|
+
- The name index is a global data structure built across **all** files.
|
|
183
|
+
- `COPY CodeRelation` is DDL and cannot run concurrently.
|
|
184
|
+
- Call resolution is lightweight (dictionary lookups) so parallelism
|
|
185
|
+
would add overhead without benefit.
|
|
186
|
+
|
|
187
|
+
#### Schema setup — single-threaded
|
|
188
|
+
|
|
189
|
+
When the database is empty, `ensure_schema()` runs the schema file
|
|
190
|
+
(`schema.cypher`) to create all node tables (`File`, `Function`,
|
|
191
|
+
`Method`, `Class`, `Struct`, `Interface`, `Trait`, `Impl`, …) and
|
|
192
|
+
the relationship table group `CodeRelation` with all permitted
|
|
193
|
+
`FROM→TO` pairs.
|
|
194
|
+
|
|
195
|
+
This runs once at the start of `--index`, before any other phases.
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
## Concurrency model summary
|
|
200
|
+
|
|
201
|
+
| Phase | Threads | DB mode | Notes |
|
|
202
|
+
|---|---|---|---|
|
|
203
|
+
| File discovery | 1 (main) | — | I/O walk, no DB |
|
|
204
|
+
| Analysis / parsing | `--workers` (ThreadPoolExecutor) | — | Thread-local parsers, no DB |
|
|
205
|
+
| Node ingestion | `--workers` (ThreadPoolExecutor) | `enable_multi_writes=true` | Each worker opens its own `Connection` |
|
|
206
|
+
| Definition edges | 1 (main) | Regular | `COPY CodeRelation` is DDL |
|
|
207
|
+
| Call resolution + edges | 1 (main) | Regular | Single-pass name index, then DDL COPY |
|
|
208
|
+
| Queries | 1 (main) | Read-only | Single `MATCH` queries |
|
|
209
|
+
|
|
210
|
+
`--workers` defaults to `min(32, cpu_count() + 4)`. It controls both
|
|
211
|
+
the analysis threads (phase 2) and the ingest threads (phase 3). When
|
|
212
|
+
`--workers=1`, phases 2 and 3 also run single-threaded (the
|
|
213
|
+
`ThreadPoolExecutor` is bypassed and the tqdm progress shows chunks
|
|
214
|
+
instead of per-file).
|
|
215
|
+
|
|
216
|
+
The entire tool is **single-process** — it uses Python threads only,
|
|
217
|
+
never `multiprocessing`. Threads are appropriate because tree-sitter
|
|
218
|
+
parsing is CPU-bound C extension work (the GIL is released) and the
|
|
219
|
+
ladybug client library is also I/O-bound on IPC with the database
|
|
220
|
+
process.
|
|
221
|
+
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
## Usage
|
|
225
|
+
|
|
226
|
+
### Indexing
|
|
227
|
+
|
|
228
|
+
```
|
|
229
|
+
uv run python3 main.py --index [PATH ...] [--language LANG] [--workers N] [--db DB] [--schema SCHEMA]
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
- `PATH` — one or more files or directories (default: current directory).
|
|
233
|
+
- `--language` / `-l` — restrict to a single language (default: all installed).
|
|
234
|
+
- `--workers` — thread count for analysis and node ingestion (default: CPU-based).
|
|
235
|
+
- `--db` / `-d` — Ladybug database file (default: `test.db`).
|
|
236
|
+
- `--schema` / `-s` — schema `.cypher` file (default: `schema.cypher` alongside `main.py`).
|
|
237
|
+
|
|
238
|
+
Example:
|
|
239
|
+
|
|
240
|
+
```
|
|
241
|
+
$ uv run python3 main.py --index icebug-format --language python --workers 4
|
|
242
|
+
Created schema from .../lscope/schema.cypher
|
|
243
|
+
Analyzing: 100%|████████████| 9/9 [00:00<00:00, 10.12 file/s]
|
|
244
|
+
Ingesting: 100%|████████████| 4/4 [00:00<00:00, 8.54 chunk/s]
|
|
245
|
+
|
|
246
|
+
Ingested 9 file(s), 90 semantic node(s), and 149 resolved call(s) into test.db using 4 analysis thread(s)
|
|
247
|
+
python: 9 file(s)
|
|
248
|
+
|
|
249
|
+
$ du -sh test.db
|
|
250
|
+
832K test.db
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
### Searching
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
uv run python3 main.py --find-functions REGEX [--db DB]
|
|
257
|
+
uv run python3 main.py --find-callers NAME [--db DB]
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
Example:
|
|
261
|
+
|
|
262
|
+
```
|
|
263
|
+
$ uv run python3 main.py --find-functions 'main'
|
|
264
|
+
Functions matching /main/:
|
|
265
|
+
shape: (4, 5)
|
|
266
|
+
┌──────┬──────────┬──────────────────────┬────────────┬──────────┐
|
|
267
|
+
│ name ┆ kind ┆ file_path ┆ start_line ┆ end_line │
|
|
268
|
+
╞══════╪══════════╪══════════════════════╪════════════╪══════════╡
|
|
269
|
+
│ main ┆ Function ┆ .../verify_edges.py ┆ 88 ┆ 114 │
|
|
270
|
+
│ ... ┆ ... ┆ ... ┆ ... ┆ ... │
|
|
271
|
+
└──────┴──────────┴──────────────────────┴────────────┴──────────┘
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
```
|
|
275
|
+
$ uv run python3 main.py --find-callers 'format_gb'
|
|
276
|
+
Callers of 'format_gb':
|
|
277
|
+
shape: (2, 6)
|
|
278
|
+
┌──────────────────────┬─────────────┬───────────┬───────────┬────────────┬──────────────────┐
|
|
279
|
+
│ caller ┆ caller_kind ┆ file_path ┆ callee ┆ confidence ┆ reason │
|
|
280
|
+
╞══════════════════════╪═════════════╪═══════════╪═══════════╪════════════╪══════════════════╡
|
|
281
|
+
│ default_memory_limit ┆ Function ┆ ... ┆ format_gb ┆ 1.0 ┆ call at ...:129 │
|
|
282
|
+
│ parse_memory_limit ┆ Function ┆ ... ┆ format_gb ┆ 1.0 ┆ call at ...:139 │
|
|
283
|
+
└──────────────────────┴─────────────┴───────────┴───────────┴────────────┴──────────────────┘
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
### Schema only
|
|
287
|
+
|
|
288
|
+
```
|
|
289
|
+
uv run python3 main.py --schema-only --schema SCHEMA --db DB
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
Applies the schema file to the database without indexing any files.
|
|
293
|
+
Useful for preparing an empty database ahead of time.
|
|
294
|
+
|
|
295
|
+
---
|
|
296
|
+
|
|
297
|
+
## Supported languages
|
|
298
|
+
|
|
299
|
+
Python, JavaScript, TypeScript, Java, C#, C++, Go, Rust, Swift, and
|
|
300
|
+
Kotlin — whenever the corresponding tree-sitter grammar package is
|
|
301
|
+
installed.
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
305
|
+
## Schema
|
|
306
|
+
|
|
307
|
+
The full schema is defined in [`schema.cypher`](./schema.cypher). It
|
|
308
|
+
defines node tables for:
|
|
309
|
+
|
|
310
|
+
- **File**, **Folder** — filesystem hierarchy.
|
|
311
|
+
- **Function**, **Method** — callable declarations.
|
|
312
|
+
- **Class**, **Struct**, **Interface**, **Trait**, **Impl** — type/container
|
|
313
|
+
declarations.
|
|
314
|
+
- **Enum**, **Property**, **CodeElement** — additional code entities.
|
|
315
|
+
- **Route**, **Tool** — framework-level metadata (routes, external tools).
|
|
316
|
+
- **Community**, **Process** — higher-level grouping nodes (for future
|
|
317
|
+
analysis passes).
|
|
318
|
+
|
|
319
|
+
All relationships are stored in a single `CodeRelation` relationship
|
|
320
|
+
table group, with a `type` property indicating the relationship kind
|
|
321
|
+
(e.g. `CALLS`, `DEFINES`, `HAS_METHOD`, `CONTAINS`, `IMPORTS`).
|
lscope-0.1.0/README.md
ADDED
|
@@ -0,0 +1,298 @@
|
|
|
1
|
+
# lscope — semantic code graph (via tree-sitter) on Ladybug
|
|
2
|
+
|
|
3
|
+
lscope parses source files with tree-sitter, extracts a semantic graph
|
|
4
|
+
(files, classes, functions, methods, calls), and stores it in a
|
|
5
|
+
[Ladybug](https://github.com/thatguyfrombb/ladybug) database. It then
|
|
6
|
+
supports lightweight code-intelligence queries: find functions by name
|
|
7
|
+
pattern, or find callers of a given function.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Pipeline phases
|
|
12
|
+
|
|
13
|
+
When invoked with `--index`, the tool runs through several phases.
|
|
14
|
+
Some are parallel, some are single-threaded. The diagram below shows
|
|
15
|
+
the data flow and concurrency model:
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
19
|
+
│ 1. File Discovery (single-threaded) │
|
|
20
|
+
│ Walk targets, match extensions, collect file list │
|
|
21
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
22
|
+
│
|
|
23
|
+
▼
|
|
24
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
25
|
+
│ 2. Analysis / Parsing (parallel — ThreadPoolExecutor) │
|
|
26
|
+
│ Each file is read + tree-sitter parsed in a worker thread. │
|
|
27
|
+
│ Every thread has its own thread-local LanguageRegistry │
|
|
28
|
+
│ (parser instances). │
|
|
29
|
+
│ Progress: tqdm "Analyzing" bar across all files. │
|
|
30
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
31
|
+
│
|
|
32
|
+
▼
|
|
33
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
34
|
+
│ 3. Node Ingestion (parallel when workers > 1) │
|
|
35
|
+
│ Analyses are split round-robin into worker chunks. │
|
|
36
|
+
│ Each chunk runs in its own thread, opening its own │
|
|
37
|
+
│ ladybug Connection with enable_multi_writes=True. │
|
|
38
|
+
│ Nodes are bulk-inserted via UNWIND + MERGE per label │
|
|
39
|
+
│ in a single transaction per chunk. │
|
|
40
|
+
│ Progress: tqdm "Ingesting" bar across chunks. │
|
|
41
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
42
|
+
│
|
|
43
|
+
▼
|
|
44
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
45
|
+
│ 4. Definition Edges (single-threaded) │
|
|
46
|
+
│ CONTAINS / DEFINES / HAS_METHOD edges inserted via │
|
|
47
|
+
│ COPY CodeRelation (DDL — cannot run concurrently with │
|
|
48
|
+
│ active write transactions from step 3). │
|
|
49
|
+
│ Runs on the main connection after all workers finish. │
|
|
50
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
51
|
+
│
|
|
52
|
+
▼
|
|
53
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
54
|
+
│ 5. Call Resolution & Edges (single-threaded) │
|
|
55
|
+
│ Build a global name index across all analyzed files. │
|
|
56
|
+
│ Resolve each call expression to a declaration by name, │
|
|
57
|
+
│ preferring same-file matches. Bulk-insert CALLS edges │
|
|
58
|
+
│ via COPY CodeRelation on the main connection. │
|
|
59
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
When running a **query** (`--find-functions` or `--find-callers`), the
|
|
63
|
+
pipeline is trivial: a single read-only connection is opened and the
|
|
64
|
+
query executes on the main thread.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
### Phase details
|
|
69
|
+
|
|
70
|
+
#### 1. File discovery — single-threaded
|
|
71
|
+
|
|
72
|
+
`iter_source_files()` walks the given paths (files or directories).
|
|
73
|
+
Directories are recursed with `os.walk`, skipping common ignorable
|
|
74
|
+
directories (`.git`, `__pycache__`, `node_modules`, `.venv`, …).
|
|
75
|
+
Files are matched against the known extension map from the installed
|
|
76
|
+
tree-sitter grammars. If `--language` is given, only files of that
|
|
77
|
+
language are kept.
|
|
78
|
+
|
|
79
|
+
This runs entirely in the main thread — it is I/O-bound on `os.walk`
|
|
80
|
+
and has no heavy computation.
|
|
81
|
+
|
|
82
|
+
#### 2. Analysis / parsing — parallel (ThreadPoolExecutor)
|
|
83
|
+
|
|
84
|
+
Each source file is read from disk and parsed by a tree-sitter `Parser`.
|
|
85
|
+
The parser is obtained from a **thread-local** `LanguageRegistry`
|
|
86
|
+
(`_worker_state.registry`) so each thread creates its own parser
|
|
87
|
+
instances — tree-sitter parsers are not thread-safe for concurrent
|
|
88
|
+
use from multiple threads.
|
|
89
|
+
|
|
90
|
+
The `analyze_path()` function:
|
|
91
|
+
1. Determines the language from the file extension.
|
|
92
|
+
2. Reads the full file into a string.
|
|
93
|
+
3. Parses it with the appropriate tree-sitter grammar.
|
|
94
|
+
4. Walks the CST to extract:
|
|
95
|
+
- **Containers** — classes, structs, interfaces, traits, impl blocks.
|
|
96
|
+
- **Functions / methods** — with their owning container.
|
|
97
|
+
- **Call expressions** — recording caller, callee name, and source location.
|
|
98
|
+
|
|
99
|
+
No database calls happen in this phase. All extracted data is returned
|
|
100
|
+
as plain dicts for the next phase.
|
|
101
|
+
|
|
102
|
+
#### 3. Node ingestion — parallel (ThreadPoolExecutor, multi-write DB)
|
|
103
|
+
|
|
104
|
+
`ingest_analyses_parallel()` splits the list of analyses into
|
|
105
|
+
`workers` chunks (round-robin). Each chunk is processed by a worker
|
|
106
|
+
thread that:
|
|
107
|
+
|
|
108
|
+
1. Opens its own `ladybug.Connection` to the shared database (which was
|
|
109
|
+
opened with `enable_multi_writes=True`).
|
|
110
|
+
2. Calls `_collect_file_data()` to organise extracted symbols by label.
|
|
111
|
+
3. Bulk-inserts all nodes for that chunk:
|
|
112
|
+
- **Files** — `UNWIND … MERGE (f:File …)`.
|
|
113
|
+
- **Symbols** — `UNWIND … MERGE (n:{label} …)` per label
|
|
114
|
+
(Function, Method, Class, Struct, …).
|
|
115
|
+
- All wrapped in a single `BEGIN TRANSACTION … COMMIT` per chunk.
|
|
116
|
+
4. Returns definition edges for deferred insertion.
|
|
117
|
+
|
|
118
|
+
**Why parallel?** Node insertion is the bulk of the write work and
|
|
119
|
+
Ladybug's multi-write mode permits concurrent write transactions from
|
|
120
|
+
different connections. This gives a significant speedup for large
|
|
121
|
+
codebases.
|
|
122
|
+
|
|
123
|
+
**Why not parallel for everything?** Step 4 (definition edges) and
|
|
124
|
+
step 5 (call edges) use `COPY CodeRelation`, which is a DDL statement
|
|
125
|
+
and cannot run concurrently with other write transactions.
|
|
126
|
+
|
|
127
|
+
#### 4. Definition edges — single-threaded
|
|
128
|
+
|
|
129
|
+
After all worker threads finish, the main thread inserts edges that
|
|
130
|
+
represent:
|
|
131
|
+
|
|
132
|
+
- **CONTAINS** — file contains a function/class/etc.
|
|
133
|
+
- **DEFINES** — a file defines a top-level symbol.
|
|
134
|
+
- **HAS_METHOD** — a container (class/struct/trait/impl) has a method.
|
|
135
|
+
|
|
136
|
+
Edges are grouped by `(source_label, target_label)`, written to a
|
|
137
|
+
temporary Parquet file per group, and bulk-loaded with
|
|
138
|
+
`COPY CodeRelation FROM '…' (FROM='…', TO='…')`.
|
|
139
|
+
|
|
140
|
+
This runs single-threaded because `COPY` on the `CodeRelation`
|
|
141
|
+
relationship table group is a DDL-level operation that requires
|
|
142
|
+
exclusive write access.
|
|
143
|
+
|
|
144
|
+
#### 5. Call resolution & edges — single-threaded
|
|
145
|
+
|
|
146
|
+
`ingest_calls()` builds a global dictionary mapping each declared
|
|
147
|
+
function/method name to its symbol dict(s), then iterates over every
|
|
148
|
+
call expression from every file:
|
|
149
|
+
|
|
150
|
+
1. Looks up the callee name in the name index.
|
|
151
|
+
2. Prefers a declaration in the **same file** (for local disambiguation).
|
|
152
|
+
3. Falls back to the first candidate if there is no same-file match.
|
|
153
|
+
4. Assigns `confidence = 1.0` for unique matches, `0.7` when multiple
|
|
154
|
+
candidates exist (ambiguous name).
|
|
155
|
+
5. Bulk-inserts all `CALLS` edges via `_ingest_chunk_edges()` — again
|
|
156
|
+
using the DDL `COPY` path.
|
|
157
|
+
|
|
158
|
+
This must be single-threaded because:
|
|
159
|
+
- The name index is a global data structure built across **all** files.
|
|
160
|
+
- `COPY CodeRelation` is DDL and cannot run concurrently.
|
|
161
|
+
- Call resolution is lightweight (dictionary lookups) so parallelism
|
|
162
|
+
would add overhead without benefit.
|
|
163
|
+
|
|
164
|
+
#### Schema setup — single-threaded
|
|
165
|
+
|
|
166
|
+
When the database is empty, `ensure_schema()` runs the schema file
|
|
167
|
+
(`schema.cypher`) to create all node tables (`File`, `Function`,
|
|
168
|
+
`Method`, `Class`, `Struct`, `Interface`, `Trait`, `Impl`, …) and
|
|
169
|
+
the relationship table group `CodeRelation` with all permitted
|
|
170
|
+
`FROM→TO` pairs.
|
|
171
|
+
|
|
172
|
+
This runs once at the start of `--index`, before any other phases.
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Concurrency model summary
|
|
177
|
+
|
|
178
|
+
| Phase | Threads | DB mode | Notes |
|
|
179
|
+
|---|---|---|---|
|
|
180
|
+
| File discovery | 1 (main) | — | I/O walk, no DB |
|
|
181
|
+
| Analysis / parsing | `--workers` (ThreadPoolExecutor) | — | Thread-local parsers, no DB |
|
|
182
|
+
| Node ingestion | `--workers` (ThreadPoolExecutor) | `enable_multi_writes=true` | Each worker opens its own `Connection` |
|
|
183
|
+
| Definition edges | 1 (main) | Regular | `COPY CodeRelation` is DDL |
|
|
184
|
+
| Call resolution + edges | 1 (main) | Regular | Single-pass name index, then DDL COPY |
|
|
185
|
+
| Queries | 1 (main) | Read-only | Single `MATCH` queries |
|
|
186
|
+
|
|
187
|
+
`--workers` defaults to `min(32, cpu_count() + 4)`. It controls both
|
|
188
|
+
the analysis threads (phase 2) and the ingest threads (phase 3). When
|
|
189
|
+
`--workers=1`, phases 2 and 3 also run single-threaded (the
|
|
190
|
+
`ThreadPoolExecutor` is bypassed and the tqdm progress shows chunks
|
|
191
|
+
instead of per-file).
|
|
192
|
+
|
|
193
|
+
The entire tool is **single-process** — it uses Python threads only,
|
|
194
|
+
never `multiprocessing`. Threads are appropriate because tree-sitter
|
|
195
|
+
parsing is CPU-bound C extension work (the GIL is released) and the
|
|
196
|
+
ladybug client library is also I/O-bound on IPC with the database
|
|
197
|
+
process.
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Usage
|
|
202
|
+
|
|
203
|
+
### Indexing
|
|
204
|
+
|
|
205
|
+
```
|
|
206
|
+
uv run python3 main.py --index [PATH ...] [--language LANG] [--workers N] [--db DB] [--schema SCHEMA]
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
- `PATH` — one or more files or directories (default: current directory).
|
|
210
|
+
- `--language` / `-l` — restrict to a single language (default: all installed).
|
|
211
|
+
- `--workers` — thread count for analysis and node ingestion (default: CPU-based).
|
|
212
|
+
- `--db` / `-d` — Ladybug database file (default: `test.db`).
|
|
213
|
+
- `--schema` / `-s` — schema `.cypher` file (default: `schema.cypher` alongside `main.py`).
|
|
214
|
+
|
|
215
|
+
Example:
|
|
216
|
+
|
|
217
|
+
```
|
|
218
|
+
$ uv run python3 main.py --index icebug-format --language python --workers 4
|
|
219
|
+
Created schema from .../lscope/schema.cypher
|
|
220
|
+
Analyzing: 100%|████████████| 9/9 [00:00<00:00, 10.12 file/s]
|
|
221
|
+
Ingesting: 100%|████████████| 4/4 [00:00<00:00, 8.54 chunk/s]
|
|
222
|
+
|
|
223
|
+
Ingested 9 file(s), 90 semantic node(s), and 149 resolved call(s) into test.db using 4 analysis thread(s)
|
|
224
|
+
python: 9 file(s)
|
|
225
|
+
|
|
226
|
+
$ du -sh test.db
|
|
227
|
+
832K test.db
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
### Searching
|
|
231
|
+
|
|
232
|
+
```
|
|
233
|
+
uv run python3 main.py --find-functions REGEX [--db DB]
|
|
234
|
+
uv run python3 main.py --find-callers NAME [--db DB]
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
Example:
|
|
238
|
+
|
|
239
|
+
```
|
|
240
|
+
$ uv run python3 main.py --find-functions 'main'
|
|
241
|
+
Functions matching /main/:
|
|
242
|
+
shape: (4, 5)
|
|
243
|
+
┌──────┬──────────┬──────────────────────┬────────────┬──────────┐
|
|
244
|
+
│ name ┆ kind ┆ file_path ┆ start_line ┆ end_line │
|
|
245
|
+
╞══════╪══════════╪══════════════════════╪════════════╪══════════╡
|
|
246
|
+
│ main ┆ Function ┆ .../verify_edges.py ┆ 88 ┆ 114 │
|
|
247
|
+
│ ... ┆ ... ┆ ... ┆ ... ┆ ... │
|
|
248
|
+
└──────┴──────────┴──────────────────────┴────────────┴──────────┘
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
```
|
|
252
|
+
$ uv run python3 main.py --find-callers 'format_gb'
|
|
253
|
+
Callers of 'format_gb':
|
|
254
|
+
shape: (2, 6)
|
|
255
|
+
┌──────────────────────┬─────────────┬───────────┬───────────┬────────────┬──────────────────┐
|
|
256
|
+
│ caller ┆ caller_kind ┆ file_path ┆ callee ┆ confidence ┆ reason │
|
|
257
|
+
╞══════════════════════╪═════════════╪═══════════╪═══════════╪════════════╪══════════════════╡
|
|
258
|
+
│ default_memory_limit ┆ Function ┆ ... ┆ format_gb ┆ 1.0 ┆ call at ...:129 │
|
|
259
|
+
│ parse_memory_limit ┆ Function ┆ ... ┆ format_gb ┆ 1.0 ┆ call at ...:139 │
|
|
260
|
+
└──────────────────────┴─────────────┴───────────┴───────────┴────────────┴──────────────────┘
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
### Schema only
|
|
264
|
+
|
|
265
|
+
```
|
|
266
|
+
uv run python3 main.py --schema-only --schema SCHEMA --db DB
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
Applies the schema file to the database without indexing any files.
|
|
270
|
+
Useful for preparing an empty database ahead of time.
|
|
271
|
+
|
|
272
|
+
---
|
|
273
|
+
|
|
274
|
+
## Supported languages
|
|
275
|
+
|
|
276
|
+
Python, JavaScript, TypeScript, Java, C#, C++, Go, Rust, Swift, and
|
|
277
|
+
Kotlin — whenever the corresponding tree-sitter grammar package is
|
|
278
|
+
installed.
|
|
279
|
+
|
|
280
|
+
---
|
|
281
|
+
|
|
282
|
+
## Schema
|
|
283
|
+
|
|
284
|
+
The full schema is defined in [`schema.cypher`](./schema.cypher). It
|
|
285
|
+
defines node tables for:
|
|
286
|
+
|
|
287
|
+
- **File**, **Folder** — filesystem hierarchy.
|
|
288
|
+
- **Function**, **Method** — callable declarations.
|
|
289
|
+
- **Class**, **Struct**, **Interface**, **Trait**, **Impl** — type/container
|
|
290
|
+
declarations.
|
|
291
|
+
- **Enum**, **Property**, **CodeElement** — additional code entities.
|
|
292
|
+
- **Route**, **Tool** — framework-level metadata (routes, external tools).
|
|
293
|
+
- **Community**, **Process** — higher-level grouping nodes (for future
|
|
294
|
+
analysis passes).
|
|
295
|
+
|
|
296
|
+
All relationships are stored in a single `CodeRelation` relationship
|
|
297
|
+
table group, with a `type` property indicating the relationship kind
|
|
298
|
+
(e.g. `CALLS`, `DEFINES`, `HAS_METHOD`, `CONTAINS`, `IMPORTS`).
|