ph-code-graph 0.10.0__tar.gz → 0.11.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ph_code_graph-0.11.0/PKG-INFO +573 -0
- ph_code_graph-0.11.0/README.md +544 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/pyproject.toml +24 -7
- ph_code_graph-0.11.0/src/ph_code_graph/__init__.py +32 -0
- ph_code_graph-0.11.0/src/ph_code_graph/__main__.py +6 -0
- ph_code_graph-0.11.0/src/ph_code_graph/_config.py +135 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/src/ph_code_graph/_extract.py +54 -28
- ph_code_graph-0.11.0/src/ph_code_graph/cli.py +45 -0
- ph_code_graph-0.10.0/src/ph_code_graph/__init__.py → ph_code_graph-0.11.0/src/ph_code_graph/engine.py +241 -128
- ph_code_graph-0.11.0/src/ph_code_graph/row.py +184 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/src/ph_code_graph/skills/code-graph/SKILL.md +11 -3
- ph_code_graph-0.11.0/tests/code_graph_helpers.py +97 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/tests/test_code_graph.py +218 -179
- ph_code_graph-0.11.0/tests/test_code_graph_cli.py +255 -0
- ph_code_graph-0.11.0/tests/test_code_graph_row.py +485 -0
- ph_code_graph-0.10.0/PKG-INFO +0 -269
- ph_code_graph-0.10.0/README.md +0 -241
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/.gitignore +0 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/LICENSE +0 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/src/ph_code_graph/_store.py +0 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/src/ph_code_graph/bundle.yaml +0 -0
- {ph_code_graph-0.10.0 → ph_code_graph-0.11.0}/src/ph_code_graph/py.typed +0 -0
|
@@ -0,0 +1,573 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: ph-code-graph
|
|
3
|
+
Version: 0.11.0
|
|
4
|
+
Summary: pH plugin: `code_index` and `code_graph`, a Python-native tree-sitter code graph with line-accurate answers.
|
|
5
|
+
Project-URL: Homepage, https://github.com/chastabor/pH
|
|
6
|
+
Project-URL: Repository, https://github.com/chastabor/pH
|
|
7
|
+
Project-URL: Documentation, https://github.com/chastabor/pH/blob/main/docs/README.md
|
|
8
|
+
Project-URL: Issues, https://github.com/chastabor/pH/issues
|
|
9
|
+
Author: Charles Tabor
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: agent,code-graph,code-search,llm,tree-sitter
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: MacOS
|
|
16
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Classifier: Topic :: Software Development
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.12
|
|
24
|
+
Requires-Dist: ph-core==0.11.0
|
|
25
|
+
Requires-Dist: ph-utcp==0.11.0
|
|
26
|
+
Requires-Dist: tree-sitter-language-pack>=1.16
|
|
27
|
+
Requires-Dist: tree-sitter>=0.25
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# ph-code-graph
|
|
31
|
+
|
|
32
|
+
*`code_index` and `code_graph`: ask a codebase about its own shape, and get back
|
|
33
|
+
a `path:start-end` you can `read`.*
|
|
34
|
+
|
|
35
|
+
Two tools, a command that runs them, and a [UTCP](https://utcp.io) manual that
|
|
36
|
+
describes them. No Node, no Rust toolchain, no submodule — tree-sitter through a
|
|
37
|
+
Python wheel, the graph in stdlib `sqlite3`.
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
ph-code-graph call code_index --paths packages/ph-core --format text
|
|
41
|
+
ph-code-graph call code_graph --mode impact --query claim_slot --format text
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
The same two tools reach a model three ways: inside a [phern](https://github.com/chastabor/pH)
|
|
45
|
+
session, typed at a shell, or registered with any other harness that speaks
|
|
46
|
+
UTCP. This package needs phern for the first only.
|
|
47
|
+
|
|
48
|
+
## Installing
|
|
49
|
+
|
|
50
|
+
Python 3.12 or newer, on Linux or macOS.
|
|
51
|
+
|
|
52
|
+
**With phern, nothing to do.** `uv tool install phern` brings this package with
|
|
53
|
+
it, and the `rlm-indexed` profile mounts it.
|
|
54
|
+
|
|
55
|
+
**By itself**, for a shell or for another harness:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
uv tool install ph-code-graph # or: pip install ph-code-graph
|
|
59
|
+
ph-code-graph install # fetch the grammars now, not during a call
|
|
60
|
+
ph-code-graph status
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
That is the `ph-code-graph` command and nothing of phern's: `ph-core` (the
|
|
64
|
+
registry the tools are written against), `ph-utcp` and the two UTCP libraries it
|
|
65
|
+
names, tree-sitter and its language pack: 24 packages and about 30 MB, measured
|
|
66
|
+
in an empty environment. No model and no torch. The grammars are fetched from
|
|
67
|
+
GitHub once, by `ph-code-graph install` or by the first index that can reach it,
|
|
68
|
+
and nothing needs the network after that.
|
|
69
|
+
|
|
70
|
+
**The command arrives in 0.11.** `ph-code-graph` 0.10.0 is the plugin
|
|
71
|
+
that ran inside a phern session; it has no command and no manual.
|
|
72
|
+
|
|
73
|
+
**From a checkout**, `uv sync` puts the command in the workspace's environment
|
|
74
|
+
(`uv run ph-code-graph ...`), and `uv tool install ./packages/ph-code-graph`
|
|
75
|
+
puts it on `PATH`.
|
|
76
|
+
|
|
77
|
+
## Three ways in
|
|
78
|
+
|
|
79
|
+
### In a phern session
|
|
80
|
+
|
|
81
|
+
The `code-graph` row registers this package's manual with `ph-utcp`, and the two
|
|
82
|
+
tools appear under their own names. A profile layers the `utcp` bundle and then
|
|
83
|
+
this one; `rlm-indexed` does:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
phern --profile rlm-indexed -p "index packages/ph-core, then tell me what breaks if I change claim_slot"
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
On another profile, add the two rows:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
phern --profile llama --provider llama --model <model> \
|
|
93
|
+
--patch '{insert: [{id: utcp, name: utcp}, {id: code-graph, name: code-graph}]}' \
|
|
94
|
+
-p "what calls claim_slot?"
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Or in a profile document:
|
|
98
|
+
|
|
99
|
+
```yaml
|
|
100
|
+
- id: utcp
|
|
101
|
+
name: utcp
|
|
102
|
+
- id: code-graph
|
|
103
|
+
name: code-graph
|
|
104
|
+
config:
|
|
105
|
+
languages: [python]
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
The index is built for you: a host is started at the agent's first step and
|
|
109
|
+
indexes the row's `index` scope in the background (see "The two tools").
|
|
110
|
+
`/code-graph status` and `/code-graph install` ask the command about its
|
|
111
|
+
grammars, and `phern doctor` reports where the index is.
|
|
112
|
+
|
|
113
|
+
**Run `/code-graph install` once, before the first session.** The grammars come
|
|
114
|
+
from GitHub, and a confined agent cannot reach it: with an empty grammar cache
|
|
115
|
+
its index skips every file, and the note that tells the agent the run ended
|
|
116
|
+
names each as skipped for want of a parser.
|
|
117
|
+
|
|
118
|
+
### From a command line
|
|
119
|
+
|
|
120
|
+
To try a query by hand, or to see why one came back empty. Run it **in the tree
|
|
121
|
+
you are asking about**: the workspace is the working directory, or `--root`.
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
ph-code-graph call code_index --paths packages/ph-core --format text
|
|
125
|
+
ph-code-graph call code_graph --mode callers --query claim_slot --format text
|
|
126
|
+
ph-code-graph tools # the tools and their flags
|
|
127
|
+
ph-code-graph status # grammars, and what the index holds
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
| command | what it does |
|
|
131
|
+
|---|---|
|
|
132
|
+
| `call TOOL [--field value ...]` | run one tool; each of its fields is a flag, or pass them all as `--args-json '{...}'` |
|
|
133
|
+
| `tools` | the tools and their flags |
|
|
134
|
+
| `status` | which grammars are ready, where the index is and what it holds |
|
|
135
|
+
| `install` | fetch every grammar now |
|
|
136
|
+
| `manual` | the UTCP manual; `manual --registration` is what a harness registers |
|
|
137
|
+
| `serve --socket PATH` | stay up and answer `call`s on a unix socket |
|
|
138
|
+
|
|
139
|
+
**A call prints one JSON object** and nothing else: `{"ok": true, "value": ...,
|
|
140
|
+
"text": ...}`, or `{"ok": false, "error": {"code": ..., "message": ...}}`.
|
|
141
|
+
`value` is the tool's result as data and `text` is the same result written for a
|
|
142
|
+
model to read; `--format text` prints only that. The exit code is 0, or 1 for a
|
|
143
|
+
failure, and the failure is in the object either way.
|
|
144
|
+
|
|
145
|
+
**Settings** are the session row's own, by row id, read from three places with
|
|
146
|
+
the later winning: `$PH_HOME/tools/ph-code-graph.yaml`, then `$PH_TOOL_CONFIG`
|
|
147
|
+
(JSON), then `--config FILE`:
|
|
148
|
+
|
|
149
|
+
```yaml
|
|
150
|
+
# ~/.ph/tools/ph-code-graph.yaml
|
|
151
|
+
code-graph:
|
|
152
|
+
languages: [python]
|
|
153
|
+
index:
|
|
154
|
+
paths: [src]
|
|
155
|
+
exclude: ["**/tests/**"]
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
**Where things go** is said by five variables, none of which has to be set:
|
|
159
|
+
|
|
160
|
+
| variable | default | what the command keeps there |
|
|
161
|
+
|---|---|---|
|
|
162
|
+
| `PH_HOME` | `~/.ph` | `tools/ph-code-graph.yaml`, the settings above |
|
|
163
|
+
| `PH_CACHE` | `$XDG_CACHE_HOME/ph`, so `~/.cache/ph` | the grammars, the log, and the index when nothing else says where |
|
|
164
|
+
| `PH_TOOL_STATE` | unset | the directory for the index; `--state DIR` is the same |
|
|
165
|
+
| `PH_TOOL_CONFIG` | unset | row settings as one JSON object, for a caller that writes no file |
|
|
166
|
+
| `PH_TOOL_HOST` | unset | the socket of a `serve` to ask instead of running the call here |
|
|
167
|
+
|
|
168
|
+
See "Where things live" for each path.
|
|
169
|
+
|
|
170
|
+
### From another harness, over UTCP
|
|
171
|
+
|
|
172
|
+
Install the package where the harness runs its commands, then register what this
|
|
173
|
+
prints:
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
ph-code-graph manual --registration
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
```json
|
|
180
|
+
{"name": "ph_code_graph",
|
|
181
|
+
"call_template_type": "cli",
|
|
182
|
+
"commands": [{"command": "/path/to/python -m ph_code_graph manual"}],
|
|
183
|
+
"inherit_env_vars": ["PATH", "HOME", "...", "PH_HOME", "PH_CACHE", "PH_TOOL_STATE", "PH_TOOL_HOST"]}
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
The harness runs that command once to read the manual, and from then on runs the
|
|
187
|
+
command each tool names. With the Python UTCP client, which this package
|
|
188
|
+
installs:
|
|
189
|
+
|
|
190
|
+
```python
|
|
191
|
+
import asyncio, json, subprocess
|
|
192
|
+
|
|
193
|
+
from utcp.utcp_client import UtcpClient
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
async def main() -> None:
|
|
197
|
+
registration = json.loads(
|
|
198
|
+
subprocess.run(
|
|
199
|
+
["ph-code-graph", "manual", "--registration"], check=True, capture_output=True
|
|
200
|
+
).stdout
|
|
201
|
+
)
|
|
202
|
+
client = await UtcpClient.create(config={"manual_call_templates": [registration]})
|
|
203
|
+
await client.call_tool("ph_code_graph.code_index", {"paths": ["src"]})
|
|
204
|
+
found = await client.call_tool(
|
|
205
|
+
"ph_code_graph.code_graph", {"mode": "callers", "query": "claim_slot"}
|
|
206
|
+
)
|
|
207
|
+
print(found["text"] if found["ok"] else found["error"])
|
|
208
|
+
|
|
209
|
+
|
|
210
|
+
asyncio.run(main())
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
What a harness needs to know:
|
|
214
|
+
|
|
215
|
+
- **The tools are `ph_code_graph.code_index` and `ph_code_graph.code_graph`.** A
|
|
216
|
+
client prefixes each tool with the manual's name.
|
|
217
|
+
- **The registration names this install.** Its command is the absolute path of
|
|
218
|
+
the interpreter the package is installed in, so it needs no `PATH`, and it is
|
|
219
|
+
printed again after the package moves.
|
|
220
|
+
- **The workspace is the directory the command is started in.** Start the
|
|
221
|
+
harness in the tree, or have it run its commands there. A `working_dir` on
|
|
222
|
+
the registered template covers reading the manual and not the tool calls, so
|
|
223
|
+
it does not move the workspace.
|
|
224
|
+
- **Give the model `text`.** Each answer is the envelope above, parsed: `text`
|
|
225
|
+
is what a model in a phern session reads, and `value` is for code. A failure
|
|
226
|
+
is `ok: false` with a `code`; the UTCP command-line protocol ignores exit
|
|
227
|
+
codes, so the envelope is the only place it is said.
|
|
228
|
+
- **Optional arguments may be left out.** The Python client logs "Missing
|
|
229
|
+
argument" for each one and the call runs.
|
|
230
|
+
- **Nothing watches the tree.** Have the model call `code_index` again after it
|
|
231
|
+
edits; a file that has not changed is not parsed again.
|
|
232
|
+
- **One index a tree.** Without `PH_TOOL_STATE` the index is kept under
|
|
233
|
+
`$PH_CACHE`, keyed by the workspace. A harness whose agents work in separate
|
|
234
|
+
trees needs nothing more; one that wants an index per agent sets
|
|
235
|
+
`PH_TOOL_STATE` for each.
|
|
236
|
+
- **In a sandbox, three places are written**: the index's directory,
|
|
237
|
+
`$PH_CACHE/tree-sitter` and `$PH_CACHE/logs`. Nothing in the tree is. The
|
|
238
|
+
grammars come from GitHub the first time, so run `ph-code-graph install`
|
|
239
|
+
outside the sandbox; after that the tools need no network.
|
|
240
|
+
- **A host is optional.** A call takes about 0.3 s by itself. A harness that
|
|
241
|
+
wants indexing to go on in the background starts
|
|
242
|
+
`ph-code-graph serve --socket PATH --warm code_index` in the tree and sets
|
|
243
|
+
`PH_TOOL_HOST=PATH` on its calls: `code_index` then answers
|
|
244
|
+
`state: indexing` after `wait_s` and the run goes on. A call that finds
|
|
245
|
+
nobody on the socket runs by itself. Keep the path short; a unix socket's is
|
|
246
|
+
limited to about a hundred characters.
|
|
247
|
+
- **The model's instructions ship with the package**, as
|
|
248
|
+
`ph_code_graph/skills/code-graph/SKILL.md`: index first, ask the question you
|
|
249
|
+
have, then read the narrow thing.
|
|
250
|
+
|
|
251
|
+
phern's own `ph-utcp` row is one such harness, and what it adds is described
|
|
252
|
+
next.
|
|
253
|
+
|
|
254
|
+
### Where the tools run, and what bounds them
|
|
255
|
+
|
|
256
|
+
In the `ph-code-graph` command's own process — started, in a pH session, by
|
|
257
|
+
`ctx.shell` for the agent that called the tool: in that agent's workspace, under
|
|
258
|
+
that agent's sandbox. So a child agent in its own worktree indexes that worktree,
|
|
259
|
+
and the harness process never loads tree-sitter.
|
|
260
|
+
|
|
261
|
+
**`permissions-fs` does not screen these tools' reads.** Its rules govern calls
|
|
262
|
+
through a session's `ctx.fs`, and a command is not one. What bounds the command
|
|
263
|
+
is the sandbox, as it bounds `bash`.
|
|
264
|
+
|
|
265
|
+
**Read-only bars the repository, not the index.** These tools change no file of
|
|
266
|
+
the tree; they build data to probe it. So the index is kept in the calling
|
|
267
|
+
agent's scratch — `$PH_HOME/scratch/<session>/<agent>/code-graph/` — which is
|
|
268
|
+
writable under every sandbox mode, while the tree is writable only under
|
|
269
|
+
`workspace-write`. An agent that may not write a tree can still index it, and
|
|
270
|
+
cannot write it by doing so. The grammar cache and the command's log are shared
|
|
271
|
+
by every agent, under `$PH_CACHE`, and the row grants those two paths.
|
|
272
|
+
|
|
273
|
+
## The two tools
|
|
274
|
+
|
|
275
|
+
**`code_index(paths?, glob?, exclude?, forget?)`** parses each file and records
|
|
276
|
+
what it defines and references.
|
|
277
|
+
|
|
278
|
+
What it looks at when a call does not say is the row's `index` block:
|
|
279
|
+
|
|
280
|
+
```yaml
|
|
281
|
+
- id: code-graph
|
|
282
|
+
name: code-graph
|
|
283
|
+
config:
|
|
284
|
+
index:
|
|
285
|
+
paths: [packages/ph-core] # directories or files; default ["."]
|
|
286
|
+
include: ["**/*.py"] # default ["**/*"] — the language filter is the real one
|
|
287
|
+
exclude: ["**/tests/**", "**/vendor/**"]
|
|
288
|
+
on_mount: true # index this when a root session starts; default true
|
|
289
|
+
languages: [python]
|
|
290
|
+
wait_s: 15 # how long code_index waits before answering "indexing"
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
**It starts unasked, and does not hold a call up.** In a session one
|
|
294
|
+
`ph-code-graph serve` stays up for each tree, kept by `ph-utcp` for the agent
|
|
295
|
+
whose tree it is. When a root session's agent takes its first step it is started
|
|
296
|
+
and indexes the `index` scope in the background; the footer shows the count
|
|
297
|
+
moving and the agent is told when it ends. A `code_index` call waits `wait_s`
|
|
298
|
+
for its run and then answers `state: indexing` with how far it has got, and
|
|
299
|
+
`code_graph` answers meanwhile from what is indexed so far and says so.
|
|
300
|
+
`on_mount: false` leaves indexing to `code_index` calls.
|
|
301
|
+
|
|
302
|
+
**A child builds its own graph only when it has a tree it writes for keeps.** A
|
|
303
|
+
child in its parent's checkout, or one granted only to read the project — every
|
|
304
|
+
child of a read-only fan-out — asks its parent's graph through its parent's
|
|
305
|
+
host, and its results say which tree their paths belong to. A child with a
|
|
306
|
+
worktree or an overlay of its own has files that diverge, so it gets its own, at
|
|
307
|
+
its first call.
|
|
308
|
+
|
|
309
|
+
`paths` is how the row is pointed at one package of a monorepo, so a bare
|
|
310
|
+
`code_index` stays under `max_files` instead of walking everything. `exclude` is
|
|
311
|
+
read against the workspace root, so `**/tests/**` means the same thing under
|
|
312
|
+
every entry of `paths`, and a directory it covers is never entered. A call's
|
|
313
|
+
`glob` replaces `include` for that call; a call's `exclude` is *added* to the
|
|
314
|
+
row's, so a call can narrow a deployment's scope and cannot lift an exclusion
|
|
315
|
+
out of it. A file named outright in `paths` is taken whatever is excluded.
|
|
316
|
+
|
|
317
|
+
**Nothing watches the filesystem.** There is no daemon and no file watcher: the
|
|
318
|
+
index changes when `code_index` runs, and at no other moment. So after your own
|
|
319
|
+
edits, run it again — which is cheap by design, and cheap enough to be a habit.
|
|
320
|
+
|
|
321
|
+
**Incremental in two layers**, and the order matters:
|
|
322
|
+
|
|
323
|
+
1. *Does this file need opening?* Asked of **git or jj**, which already hold a
|
|
324
|
+
content-addressed tree — `ph.seams.changes`. A file the version control
|
|
325
|
+
vouches for is never read. Measured on `packages/ph-core/src/ph`: a re-index
|
|
326
|
+
of 137 unchanged files went from 0.10 s to **0.03 s**, with zero reads and
|
|
327
|
+
zero hashes, on a `tree_state` costing 7 ms for the whole 509-file repo.
|
|
328
|
+
2. *Is what I opened different?* Asked of the file's **sha256**, exactly as
|
|
329
|
+
before. Never an mtime — a checkout or a rebase moves timestamps without
|
|
330
|
+
moving content, and re-indexing because git touched them is a cost nobody
|
|
331
|
+
attributes correctly.
|
|
332
|
+
|
|
333
|
+
Layer 1 only decides what to open, so the guarantee layer 2 makes is untouched:
|
|
334
|
+
a skip is always justified by somebody's content hash, never by a clock. A tree
|
|
335
|
+
with no version control loses layer 1 and nothing else.
|
|
336
|
+
|
|
337
|
+
**`code_graph(mode, query?, …)`** answers six questions:
|
|
338
|
+
|
|
339
|
+
| `mode` | question | what comes back |
|
|
340
|
+
|---|---|---|
|
|
341
|
+
| `search` | "something about workspace tiers" | symbols ranked by FTS5 over name + docstring |
|
|
342
|
+
| `define` | "where exactly is `claim_slot`" | every definition of that name |
|
|
343
|
+
| `callers` | "what calls this" | the calling symbol **and the calling line** |
|
|
344
|
+
| `callees` | "what does this call" | the definitions it reaches |
|
|
345
|
+
| `impact` | "what breaks if I change this" | transitive callers, ring by ring |
|
|
346
|
+
| `entities` | "what is big here" | definitions by span, biggest first |
|
|
347
|
+
|
|
348
|
+
Every result carries `path:start-line-end-line`. That is the point of the
|
|
349
|
+
package: the model finds what matters in one call, then `read`s exactly that.
|
|
350
|
+
|
|
351
|
+
## Using it from the RLM
|
|
352
|
+
|
|
353
|
+
Nothing to do. Under Code Mode the model is handed one callable and a generated
|
|
354
|
+
SDK listing, and *every* registered tool is in that listing — a tool that
|
|
355
|
+
arrives through a manual included — so these arrive as
|
|
356
|
+
`await tools.code_index(...)` and `await tools.code_graph(...)` with no Code
|
|
357
|
+
Mode work in this package at all. The `rlm-indexed` profile is `rlm-stable` plus
|
|
358
|
+
this bundle and `ph-text-index`'s:
|
|
359
|
+
|
|
360
|
+
```bash
|
|
361
|
+
phern --profile rlm-indexed --provider llama --model <model> --mode tui
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
This package registers a `ph.bundles` entry point, which is what lets `ph-app`
|
|
365
|
+
compose that profile without depending on this distribution — and what makes an
|
|
366
|
+
install missing it see no `rlm-indexed` rather than one that fails at mount.
|
|
367
|
+
|
|
368
|
+
## It is a name-based graph, and it says so
|
|
369
|
+
|
|
370
|
+
A reference records the *name* it used, and `callers`/`callees` join on that
|
|
371
|
+
name. Two `register` methods in two classes are one name to this index.
|
|
372
|
+
|
|
373
|
+
Every answer that could be ambiguous carries `definitions` — how many places
|
|
374
|
+
that name is defined — and the rendered text says so outright when it is above
|
|
375
|
+
one, so the model sees the ambiguity instead of being handed one of the
|
|
376
|
+
possibilities. `mode=define` is how it disambiguates.
|
|
377
|
+
|
|
378
|
+
Resolving properly means an import graph, scope, and per-language type
|
|
379
|
+
inference. In the Rust/Node CodeGraph that is 29,708 lines of TypeScript, and
|
|
380
|
+
this package deliberately did not port it. Name-based matching answers most of
|
|
381
|
+
what an agent actually asks and reports where it cannot — which is a better
|
|
382
|
+
trade than a resolved graph for one language.
|
|
383
|
+
|
|
384
|
+
## Why Python-native, and not a wrapper
|
|
385
|
+
|
|
386
|
+
The obvious plan was to submodule the Rust/Node CodeGraph and wrap its kernel
|
|
387
|
+
with maturin. The investigation said no, for three reasons in increasing order
|
|
388
|
+
of importance:
|
|
389
|
+
|
|
390
|
+
1. **`codegraph-kernel` is a `#[napi]` crate** — `crate-type = ["cdylib"]`
|
|
391
|
+
against `napi`/`napi-derive`. maturin builds PyO3/cffi extensions; wrapping
|
|
392
|
+
it means forking the crate to rewrite its boundary.
|
|
393
|
+
2. **It is only an extractor.** Its whole export surface is `extract_file`,
|
|
394
|
+
`contract_info`, `grammar_info` and two `cfnptr` helpers — "tree-sitter
|
|
395
|
+
parse+extract with one JS boundary crossing per file", in its own words. It
|
|
396
|
+
also has a complete TS/WASM fallback, so it is an accelerator, not the core.
|
|
397
|
+
3. **The intelligence is TypeScript**: `resolution/` 29,708 lines, `extraction/`
|
|
398
|
+
25,612, `mcp/` 13,940, `graph/` 6,023, `db/` 5,475 — 112,612 against the
|
|
399
|
+
kernel's 25,242. Wrapping the kernel gets a parser and leaves the product
|
|
400
|
+
behind.
|
|
401
|
+
|
|
402
|
+
And the parser is the part Python already has. `tree-sitter-language-pack` is a
|
|
403
|
+
3.7 MB wheel with 26 languages it calls bundled (python, ts/tsx/js, rust, go,
|
|
404
|
+
java, csharp, c, cpp, ruby, php, swift, kotlin, scala, dart, lua, r, and more)
|
|
405
|
+
and 371 available. **None of them works offline on a fresh cache.** Measured
|
|
406
|
+
2026-11-11 with 1.16.2 and 1.25.0: with no network and an empty cache no parser
|
|
407
|
+
is to be had, python's included; with the network the bundled set arrives as one
|
|
408
|
+
archive from GitHub the first time a parser is asked for. After that the cache
|
|
409
|
+
is all it needs.
|
|
410
|
+
|
|
411
|
+
## Why `sqlite3` and not `pyturso`
|
|
412
|
+
|
|
413
|
+
pH's session log runs on turso, so this is the exception and the reason is
|
|
414
|
+
measured. Against a real 40 MB CodeGraph database (11,396 nodes, 36,830 edges),
|
|
415
|
+
`pyturso` served indexed lookups, joins and aggregates correctly, and then:
|
|
416
|
+
|
|
417
|
+
- **FTS5 is absent** — a `CREATE VIRTUAL TABLE … USING fts5` is invisible to
|
|
418
|
+
it, shadow tables included. That is `search`.
|
|
419
|
+
- **`Recursive CTEs are not yet supported`** — that is `impact`.
|
|
420
|
+
- **`json_extract` silently returns NULL** where SQLite returns the value:
|
|
421
|
+
`0.95` against `None`, same rows, same expression. A wrong answer with no
|
|
422
|
+
error is worse than a missing feature.
|
|
423
|
+
|
|
424
|
+
Two of the six modes here are exactly the two turso cannot serve. Stdlib
|
|
425
|
+
`sqlite3` (SQLite 3.45, FTS5 compiled in) costs no dependency and does all six.
|
|
426
|
+
If turso grows both, `_store.py` changes one import.
|
|
427
|
+
|
|
428
|
+
## Extraction: two passes, and why
|
|
429
|
+
|
|
430
|
+
- **`process()`**, the pack's intelligence layer — definitions with kinds, the
|
|
431
|
+
file's imports, docstrings and comments. What it will not give is
|
|
432
|
+
*references*: `SymbolInfo` says a function exists, never who calls it.
|
|
433
|
+
- **the tags query**, tree-sitter's own `tags.scm` per language — the data
|
|
434
|
+
GitHub's code navigation is built on. `@definition.*` and, crucially,
|
|
435
|
+
`@reference.call`.
|
|
436
|
+
|
|
437
|
+
Measured over `packages/ph-core/src/ph`: 136 files and 1.58 MiB through
|
|
438
|
+
`process()` in 219 ms, through the tags query in 108 ms. Indexing this
|
|
439
|
+
repository's core takes 2.0 s end to end including the SQLite writes, and
|
|
440
|
+
0.1 s when nothing has changed. Parsing twice is worth more than the parse it
|
|
441
|
+
saves.
|
|
442
|
+
|
|
443
|
+
Four details that are easy to get wrong, each handled once in `_extract` and
|
|
444
|
+
each pinned by a test:
|
|
445
|
+
|
|
446
|
+
- **`ProcessConfig` spans are 0-based** while tree-sitter points are 0-based and
|
|
447
|
+
every line pH shows a model is 1-based. Normalized on the way out, so nothing
|
|
448
|
+
downstream has to remember which pass a number came from. Getting this wrong
|
|
449
|
+
is an off-by-one pointer, which is what makes a model stop trusting a tool.
|
|
450
|
+
- **`SymbolInfo.doc` is always `None`.** The pack does not populate it —
|
|
451
|
+
measured against a file full of docstrings. Trusting it would have shipped an
|
|
452
|
+
index with an empty `doc` column and a `search` mode matching names only. The
|
|
453
|
+
prose is on two other channels: `result.docstrings`, which carries the pack's
|
|
454
|
+
own `associated_item`, and `result.comments` for doc comments (Rust `///`,
|
|
455
|
+
TypeScript block comments, Go `//`). Those have no association, so the rule is
|
|
456
|
+
positional and narrow — the comment's last line must be *immediately* above
|
|
457
|
+
the definition's first.
|
|
458
|
+
- **A comment node's `span.end_line` can be one past the line it occupies**,
|
|
459
|
+
because Rust's `///` node includes its trailing newline. Trusting it missed
|
|
460
|
+
the definition directly below *and* matched one a blank line away. The line
|
|
461
|
+
count comes from the comment's own text instead.
|
|
462
|
+
- **TypeScript's tags query is half the story.** Its grammar extends
|
|
463
|
+
javascript's and upstream splits the queries to match, so `class`, `function`
|
|
464
|
+
and `call` live in the *javascript* query — a `.ts` file matched against the
|
|
465
|
+
typescript query alone yields two tags and no calls. `INHERITS` fixes that.
|
|
466
|
+
|
|
467
|
+
## Per-language caveats
|
|
468
|
+
|
|
469
|
+
Both are pinned by tests, so they are known shapes rather than surprises:
|
|
470
|
+
|
|
471
|
+
- **C has definitions but no call references.** Its tags query ships no
|
|
472
|
+
`@reference.*` captures at all, so `callers`/`callees` are empty for C while
|
|
473
|
+
`search`, `define` and `entities` work.
|
|
474
|
+
- **A bare Ruby send is invisible.** `helper` without parentheses parses as an
|
|
475
|
+
identifier rather than a `call`, so only `helper()` becomes an edge.
|
|
476
|
+
|
|
477
|
+
Only `reference.call` is stored. The tags queries also emit `reference.type`,
|
|
478
|
+
`reference.class` and more; each is a real relationship, but folding them in
|
|
479
|
+
would make `callers` return every place that merely *mentions* a type.
|
|
480
|
+
|
|
481
|
+
## Where things live
|
|
482
|
+
|
|
483
|
+
- **the index, in a session**: `graph.db` in the scratch of the agent whose tree
|
|
484
|
+
it is, `$PH_HOME/scratch/<session>/<agent>/code-graph/`: the calling agent, or
|
|
485
|
+
the ancestor whose tree it shares. A child in a worktree of its own indexes its
|
|
486
|
+
own tree, so two agents on different trees never answer from each other's line
|
|
487
|
+
numbers. A resumed session has the same session and agent ids, so
|
|
488
|
+
the same index: `code_index` then re-parses only what changed. The cost is that
|
|
489
|
+
a new session builds its own rather than finding the last one's.
|
|
490
|
+
- **the index, from a shell**: `--state DIR` puts it in `DIR`, and so does
|
|
491
|
+
`$PH_TOOL_STATE` — the variable `ph-utcp` sets per call, and another harness
|
|
492
|
+
can set. With neither it is
|
|
493
|
+
`$PH_CACHE/code-graph/<digest of the workspace root>/graph.db`: the cache root
|
|
494
|
+
because it is rebuildable from the source, and keyed by root so two checkouts
|
|
495
|
+
do not answer each other's questions. That directory records its workspace,
|
|
496
|
+
and one whose workspace has been gone a week is removed on the next index.
|
|
497
|
+
- **the grammars**: `$PH_CACHE/tree-sitter`, set by the engine row when the
|
|
498
|
+
command mounts it.
|
|
499
|
+
- **the writer lock**: `graph.db.lock` beside the database. One indexing run at a
|
|
500
|
+
time per index, across every process on the workspace; a second run waits for
|
|
501
|
+
the first and then finds most of the tree current.
|
|
502
|
+
- **the command's log**: `$PH_CACHE/logs/ph-code-graph.log`. A call prints one
|
|
503
|
+
JSON object and nothing on stderr, so this is where a traceback goes.
|
|
504
|
+
|
|
505
|
+
That second one is not housekeeping. The pack keeps every grammar it uses in a
|
|
506
|
+
writable cache — its bundled ones too, which it downloads there on first use —
|
|
507
|
+
and it **fails hard** when that directory cannot be created. Its own default is
|
|
508
|
+
`$XDG_CACHE_HOME/…`, which is right on a laptop and wrong in a container with a
|
|
509
|
+
read-only `HOME`; both measured. So the row hands over the root pH already
|
|
510
|
+
designates for rebuildable artifacts, and refuses to mount with one sentence —
|
|
511
|
+
not a `pathlib` traceback — when even that is unwritable. From the command that
|
|
512
|
+
sentence is the call's answer, with the code `MOUNT_REFUSED`.
|
|
513
|
+
|
|
514
|
+
`TREE_SITTER_LANGUAGE_PACK_CACHE_DIR` still wins if an operator set it: their
|
|
515
|
+
spelling is what makes the variable mean anything.
|
|
516
|
+
|
|
517
|
+
Two consequences worth expecting. The **first** index on a cold grammar cache
|
|
518
|
+
pays for fetching the grammar: measured on `packages/ph-core/src/ph`, 9 s cold
|
|
519
|
+
against 2 s warm, and 0.1 s when nothing changed. And a command that cannot
|
|
520
|
+
reach GitHub — a confined agent's, a machine with no network — indexes nothing
|
|
521
|
+
until the cache is filled: each file is skipped, with a reason that names
|
|
522
|
+
`/code-graph install`.
|
|
523
|
+
|
|
524
|
+
Provision it on purpose rather than during someone's turn:
|
|
525
|
+
|
|
526
|
+
```
|
|
527
|
+
/code-graph status # how many grammars are ready, and where they live
|
|
528
|
+
/code-graph install # load each one now, naming any that will not
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
A **command**, not a tool — a person asks the harness to provision, and it costs
|
|
532
|
+
no model turn. In a session it runs `ph-code-graph status` or `install` and shows
|
|
533
|
+
what that said; from a shell, type those directly. `phern doctor` reports the
|
|
534
|
+
index's path, whether it is built and whether the manual was discovered, without
|
|
535
|
+
starting the command. `install` fetches all eighteen grammars in about three
|
|
536
|
+
seconds, and it runs outside the sandbox, because it needs the network.
|
|
537
|
+
|
|
538
|
+
## The skill
|
|
539
|
+
|
|
540
|
+
The package installs a `code-graph` skill, so an RLM under progressive
|
|
541
|
+
disclosure sees one line in its catalog and can read the page when it needs
|
|
542
|
+
it — index first, ask the question you actually have, then `read` the narrow
|
|
543
|
+
thing; and that a name-based graph reports ambiguity rather than resolving it.
|
|
544
|
+
|
|
545
|
+
Registered by the row rather than found by a directory scan, so it arrives
|
|
546
|
+
exactly when the tools do and leaves with them — `skills-progressive` ships an
|
|
547
|
+
empty `paths` on purpose, because a skill is something a distribution installs
|
|
548
|
+
deliberately (I7).
|
|
549
|
+
|
|
550
|
+
## Tests
|
|
551
|
+
|
|
552
|
+
`tests/test_code_graph.py` runs the real parser and the real SQLite index.
|
|
553
|
+
Three things it pins deliberately:
|
|
554
|
+
|
|
555
|
+
- **line spans against the file** — every span is checked by slicing the actual
|
|
556
|
+
source at it, not against an expected number, because a pointer that is off
|
|
557
|
+
by one still looks plausible in a diff;
|
|
558
|
+
- **the enclosing-symbol rule** — a call inside a nested closure belongs to the
|
|
559
|
+
closure, not the 40-line function around it;
|
|
560
|
+
- **incrementality** — a second index pass over an unchanged tree parses nothing
|
|
561
|
+
and a touched-but-unmodified file stays unchanged.
|
|
562
|
+
|
|
563
|
+
`tests/test_code_graph_cli.py` starts real processes, because what it holds only
|
|
564
|
+
exists between them: the command answers with the values the row answers with,
|
|
565
|
+
two processes indexing one tree report counts that add up, and a real UTCP
|
|
566
|
+
client registers the manual and calls a tool with a list argument and with
|
|
567
|
+
optional arguments left out.
|
|
568
|
+
|
|
569
|
+
`tests/test_code_graph_row.py` holds the row a session mounts: the tools keep
|
|
570
|
+
their names, values, rendered text and error codes; an agent in its own tree
|
|
571
|
+
indexes that tree; a confined agent that may not write its tree still indexes
|
|
572
|
+
it; and a fresh interpreter that mounts only the session's rows never loads
|
|
573
|
+
tree-sitter.
|