ph-code-graph 0.11.0__tar.gz → 0.13.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ph_code_graph-0.13.0/PKG-INFO +641 -0
- ph_code_graph-0.13.0/README.md +612 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/pyproject.toml +15 -12
- ph_code_graph-0.13.0/src/ph_code_graph/__init__.py +32 -0
- ph_code_graph-0.13.0/src/ph_code_graph/__main__.py +6 -0
- ph_code_graph-0.13.0/src/ph_code_graph/_config.py +142 -0
- ph_code_graph-0.13.0/src/ph_code_graph/_extract.py +695 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/src/ph_code_graph/_store.py +201 -180
- ph_code_graph-0.13.0/src/ph_code_graph/bundle.yaml +22 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/src/ph_code_graph/cli.py +8 -7
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/src/ph_code_graph/engine.py +306 -295
- ph_code_graph-0.13.0/src/ph_code_graph/row.py +195 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/src/ph_code_graph/skills/code-graph/SKILL.md +10 -4
- ph_code_graph-0.13.0/tests/code_graph_helpers.py +102 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/tests/test_code_graph.py +365 -272
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/tests/test_code_graph_cli.py +97 -37
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/tests/test_code_graph_row.py +85 -72
- ph_code_graph-0.11.0/PKG-INFO +0 -573
- ph_code_graph-0.11.0/README.md +0 -544
- ph_code_graph-0.11.0/src/ph_code_graph/__init__.py +0 -32
- ph_code_graph-0.11.0/src/ph_code_graph/__main__.py +0 -6
- ph_code_graph-0.11.0/src/ph_code_graph/_config.py +0 -135
- ph_code_graph-0.11.0/src/ph_code_graph/_extract.py +0 -602
- ph_code_graph-0.11.0/src/ph_code_graph/bundle.yaml +0 -21
- ph_code_graph-0.11.0/src/ph_code_graph/row.py +0 -184
- ph_code_graph-0.11.0/tests/code_graph_helpers.py +0 -97
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/.gitignore +0 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/LICENSE +0 -0
- {ph_code_graph-0.11.0 → ph_code_graph-0.13.0}/src/ph_code_graph/py.typed +0 -0
|
@@ -0,0 +1,641 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: ph-code-graph
|
|
3
|
+
Version: 0.13.0
|
|
4
|
+
Summary: pH plugin: `code_index` and `code_graph`, a Python-native tree-sitter code graph with line-accurate answers.
|
|
5
|
+
Project-URL: Homepage, https://github.com/chastabor/pH
|
|
6
|
+
Project-URL: Repository, https://github.com/chastabor/pH
|
|
7
|
+
Project-URL: Documentation, https://github.com/chastabor/pH/blob/main/docs/README.md
|
|
8
|
+
Project-URL: Issues, https://github.com/chastabor/pH/issues
|
|
9
|
+
Author: Charles Tabor
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: agent,code-graph,code-search,llm,tree-sitter
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: MacOS
|
|
16
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Classifier: Topic :: Software Development
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.12
|
|
24
|
+
Requires-Dist: ph-core==0.13.0
|
|
25
|
+
Requires-Dist: ph-utcp==0.13.0
|
|
26
|
+
Requires-Dist: tree-sitter-language-pack>=1.16
|
|
27
|
+
Requires-Dist: tree-sitter>=0.25
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# ph-code-graph
|
|
31
|
+
|
|
32
|
+
*`code_index` and `code_graph`: ask a codebase about its own structure, and get
|
|
33
|
+
back a `path:start-end` that you can `read`.*
|
|
34
|
+
|
|
35
|
+
This package has two tools, a command that runs them, and a
|
|
36
|
+
[UTCP](https://utcp.io) manual that describes them. It needs no Node, no Rust
|
|
37
|
+
toolchain, and no submodule. It uses tree-sitter through a Python wheel, and
|
|
38
|
+
keeps the graph in stdlib `sqlite3`.
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
ph-code-graph call code_index --paths packages/ph-core --format text
|
|
42
|
+
ph-code-graph call code_graph --mode impact --query claim_slot --format text
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
The same two tools reach a model in three ways: inside a
|
|
46
|
+
[phern](https://github.com/chastabor/pH) session, typed at a shell, or registered
|
|
47
|
+
with any other harness that supports UTCP. Only the first way needs phern.
|
|
48
|
+
|
|
49
|
+
## Installing
|
|
50
|
+
|
|
51
|
+
The package needs Python 3.12 or newer, on Linux or macOS.
|
|
52
|
+
|
|
53
|
+
**With phern, there is nothing to do.** `uv tool install phern` installs this
|
|
54
|
+
package too, and the `rlm-indexed` profile mounts it.
|
|
55
|
+
|
|
56
|
+
**Without phern**, for a shell or for another harness:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
uv tool install ph-code-graph # or: pip install ph-code-graph
|
|
60
|
+
ph-code-graph install # fetch the grammars now, not during a call
|
|
61
|
+
ph-code-graph status
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
That installs the `ph-code-graph` command and nothing from phern. The
|
|
65
|
+
dependencies are `ph-core` (the registry that the tools are written against),
|
|
66
|
+
`ph-utcp` and the two UTCP libraries that it names, and tree-sitter with its
|
|
67
|
+
language pack. That is 24 packages and about 30 MB, measured in an empty
|
|
68
|
+
environment. There is no model and no torch. The grammars are fetched from
|
|
69
|
+
GitHub one time, by `ph-code-graph install` or by the first index that can reach
|
|
70
|
+
GitHub. After that, nothing needs the network.
|
|
71
|
+
|
|
72
|
+
**The command arrives in 0.11.** `ph-code-graph` 0.10.0 is the plugin that ran
|
|
73
|
+
inside a phern session. It has no command and no manual.
|
|
74
|
+
|
|
75
|
+
**From a checkout**, `uv sync` puts the command in the environment of the
|
|
76
|
+
workspace (`uv run ph-code-graph ...`), and
|
|
77
|
+
`uv tool install ./packages/ph-code-graph` puts it on `PATH`.
|
|
78
|
+
|
|
79
|
+
## Three ways in
|
|
80
|
+
|
|
81
|
+
### In a phern session
|
|
82
|
+
|
|
83
|
+
The `code-graph` row registers the manual of this package with `ph-utcp`, and the
|
|
84
|
+
two tools appear under their own names. A profile layers the `utcp` bundle and
|
|
85
|
+
then this bundle. `rlm-indexed` does that:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
phern --profile rlm-indexed -p "index packages/ph-core, then tell me what breaks if I change claim_slot"
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
On another profile, add the two rows:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
phern --profile llama --provider llama --model <model> \
|
|
95
|
+
--patch '{insert: [{id: utcp, name: utcp}, {id: code-graph, name: code-graph}]}' \
|
|
96
|
+
-p "what calls claim_slot?"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Or add them in a profile document:
|
|
100
|
+
|
|
101
|
+
```yaml
|
|
102
|
+
- id: utcp
|
|
103
|
+
name: utcp
|
|
104
|
+
- id: code-graph
|
|
105
|
+
name: code-graph
|
|
106
|
+
config:
|
|
107
|
+
languages: [python]
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
The index is built for you. Before the first step of the agent, an indexing run
|
|
111
|
+
starts in the background over the `index` scope of the row (see "The two
|
|
112
|
+
tools"). `/code-graph status` and `/code-graph install` ask the command about its
|
|
113
|
+
grammars, and `phern doctor` reports where the index is.
|
|
114
|
+
|
|
115
|
+
**Run `/code-graph install` one time, before the first session.** The grammars
|
|
116
|
+
come from GitHub, and a confined agent cannot reach GitHub. With an empty grammar
|
|
117
|
+
cache, the index of such an agent skips every file. The note that tells the agent
|
|
118
|
+
that the run ended then names each file as skipped because no parser was
|
|
119
|
+
available.
|
|
120
|
+
|
|
121
|
+
### From a command line
|
|
122
|
+
|
|
123
|
+
Use the command line to try a query by hand, or to see why a query returned
|
|
124
|
+
nothing. Run the command **in the tree that you ask about**. The workspace is the
|
|
125
|
+
working directory, or the value of `--root`.
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
ph-code-graph call code_index --paths packages/ph-core --format text
|
|
129
|
+
ph-code-graph call code_graph --mode callers --query claim_slot --format text
|
|
130
|
+
ph-code-graph tools # the tools and their flags
|
|
131
|
+
ph-code-graph status # grammars, and what the index holds
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
| Command | What it does |
|
|
135
|
+
|---|---|
|
|
136
|
+
| `call TOOL [--field value ...]` | Runs one tool. Each field of the tool is a flag, or pass all fields as `--args-json '{...}'` |
|
|
137
|
+
| `tools` | Lists the tools and their flags |
|
|
138
|
+
| `status` | Shows which grammars are ready, where the index is, and what it holds |
|
|
139
|
+
| `install` | Fetches every grammar now |
|
|
140
|
+
| `manual` | Prints the UTCP manual. `manual --registration` prints what a harness registers |
|
|
141
|
+
|
|
142
|
+
**A call prints one JSON object** and nothing else: `{"ok": true, "value": ...,
|
|
143
|
+
"text": ...}`, or `{"ok": false, "error": {"code": ..., "message": ...}}`.
|
|
144
|
+
`value` is the result of the tool as data. `text` is the same result, written
|
|
145
|
+
for a model to read. `--format text` prints only `text`. The exit code is 0, or
|
|
146
|
+
1 for a failure. A failure is also in the object in both cases.
|
|
147
|
+
|
|
148
|
+
**Every call is one command.** No process stays running between calls, and no
|
|
149
|
+
process listens on a socket. Two flags of `call` are for a run that takes a long
|
|
150
|
+
time:
|
|
151
|
+
|
|
152
|
+
- **`--progress`** reports the progress of the run on stderr as JSON Lines, one
|
|
153
|
+
JSON object on each line. While the run works, it writes
|
|
154
|
+
`{"state":"working","done":4,"total":8,"what":"files"}` at most every 0.5 s.
|
|
155
|
+
It then writes exactly one last line:
|
|
156
|
+
- `"state":"done"`, with `changed` and a `note`.
|
|
157
|
+
- `"state":"busy"`, when another run held the index and this run did nothing.
|
|
158
|
+
- `"state":"failed"`, with a `code` and a `failure`.
|
|
159
|
+
|
|
160
|
+
If the stream ends with no last line, the command was killed or it died.
|
|
161
|
+
stdout does not change: it carries one object.
|
|
162
|
+
- **`--wait SECONDS`** is how long the call waits for another run that holds the
|
|
163
|
+
index. It overrides the `wait_s` of the row. `--wait 0` answers immediately.
|
|
164
|
+
|
|
165
|
+
**Settings** are the settings of the session row, under the row id. They are read
|
|
166
|
+
from three places, and a later place overrides an earlier one:
|
|
167
|
+
`$PH_HOME/tools/ph-code-graph.yaml`, then `$PH_TOOL_CONFIG` (JSON), then
|
|
168
|
+
`--config FILE`:
|
|
169
|
+
|
|
170
|
+
```yaml
|
|
171
|
+
# ~/.ph/tools/ph-code-graph.yaml
|
|
172
|
+
code-graph:
|
|
173
|
+
languages: [python]
|
|
174
|
+
index:
|
|
175
|
+
paths: [src]
|
|
176
|
+
exclude: ["**/tests/**"]
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
**Four variables control where things go.** None of them has to be set:
|
|
180
|
+
|
|
181
|
+
| Variable | Default | What the command keeps there |
|
|
182
|
+
|---|---|---|
|
|
183
|
+
| `PH_HOME` | `~/.ph` | `tools/ph-code-graph.yaml`, the settings above |
|
|
184
|
+
| `PH_CACHE` | `$XDG_CACHE_HOME/ph`, which is usually `~/.cache/ph` | The grammars, the log, and the index when nothing else says where |
|
|
185
|
+
| `PH_TOOL_STATE` | unset | The directory for the index. `--state DIR` does the same |
|
|
186
|
+
| `PH_TOOL_CONFIG` | unset | The row settings as one JSON object, for a caller that writes no file |
|
|
187
|
+
|
|
188
|
+
See "Where things live" for each path.
|
|
189
|
+
|
|
190
|
+
### From another harness, over UTCP
|
|
191
|
+
|
|
192
|
+
Install the package where the harness runs its commands. Then register what this
|
|
193
|
+
command prints:
|
|
194
|
+
|
|
195
|
+
```bash
|
|
196
|
+
ph-code-graph manual --registration
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
```json
|
|
200
|
+
{"name": "ph_code_graph",
|
|
201
|
+
"call_template_type": "cli",
|
|
202
|
+
"commands": [{"command": "/path/to/python -P -m ph_code_graph manual"}],
|
|
203
|
+
"inherit_env_vars": ["PATH", "HOME", "...", "PH_HOME", "PH_CACHE", "PH_TOOL_STATE"]}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
The harness runs that command one time to read the manual. After that, it runs
|
|
207
|
+
the command that each tool names. This example uses the Python UTCP client,
|
|
208
|
+
which this package installs:
|
|
209
|
+
|
|
210
|
+
```python
|
|
211
|
+
import asyncio, json, subprocess
|
|
212
|
+
|
|
213
|
+
from utcp.utcp_client import UtcpClient
|
|
214
|
+
|
|
215
|
+
|
|
216
|
+
async def main() -> None:
|
|
217
|
+
registration = json.loads(
|
|
218
|
+
subprocess.run(
|
|
219
|
+
["ph-code-graph", "manual", "--registration"], check=True, capture_output=True
|
|
220
|
+
).stdout
|
|
221
|
+
)
|
|
222
|
+
client = await UtcpClient.create(config={"manual_call_templates": [registration]})
|
|
223
|
+
await client.call_tool("ph_code_graph.code_index", {"paths": ["src"]})
|
|
224
|
+
found = await client.call_tool(
|
|
225
|
+
"ph_code_graph.code_graph", {"mode": "callers", "query": "claim_slot"}
|
|
226
|
+
)
|
|
227
|
+
print(found["text"] if found["ok"] else found["error"])
|
|
228
|
+
|
|
229
|
+
|
|
230
|
+
asyncio.run(main())
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
What a harness needs to know:
|
|
234
|
+
|
|
235
|
+
- **The tools are `ph_code_graph.code_index` and `ph_code_graph.code_graph`.** A
|
|
236
|
+
client puts the name of the manual before each tool name.
|
|
237
|
+
- **The registration names this install.** Its command is the absolute path of
|
|
238
|
+
the interpreter in which the package is installed, so it needs no `PATH`.
|
|
239
|
+
Print the registration again after the package moves. `-P` keeps the working
|
|
240
|
+
directory off the module path, so the command imports no file from the tree.
|
|
241
|
+
- **The workspace is the directory in which the command starts.** Start the
|
|
242
|
+
harness in the tree, or make it run its commands there. A `working_dir` on the
|
|
243
|
+
registered template applies to reading the manual and not to the tool calls,
|
|
244
|
+
so it does not move the workspace.
|
|
245
|
+
- **Give the model `text`.** Each answer is the envelope above, parsed. `text` is
|
|
246
|
+
what a model in a phern session reads, and `value` is for code. A failure is
|
|
247
|
+
`ok: false` with a `code`. The UTCP command-line protocol ignores exit codes,
|
|
248
|
+
so the envelope is the only place that reports a failure.
|
|
249
|
+
- **Optional arguments can be left out.** The Python client logs "Missing
|
|
250
|
+
argument" for each one, and the call still runs.
|
|
251
|
+
- **Nothing watches the tree.** Make the model call `code_index` again after it
|
|
252
|
+
edits. A file that has not changed is not parsed again.
|
|
253
|
+
- **One index for each tree.** Without `PH_TOOL_STATE`, the index is kept under
|
|
254
|
+
`$PH_CACHE`, keyed by the workspace. A harness whose agents work in separate
|
|
255
|
+
trees needs nothing more. A harness that wants an index for each agent sets
|
|
256
|
+
`PH_TOOL_STATE` for each agent.
|
|
257
|
+
- **In a sandbox, the command writes three places**: the directory of the index,
|
|
258
|
+
`$PH_CACHE/tree-sitter`, and `$PH_CACHE/logs`. It writes nothing in the tree.
|
|
259
|
+
The grammars come from GitHub the first time, so run `ph-code-graph install`
|
|
260
|
+
outside the sandbox. After that the tools need no network.
|
|
261
|
+
- **One indexing run at a time.** A `code_index` call is one command, and it
|
|
262
|
+
finishes before it answers. A call that is made while another run holds the
|
|
263
|
+
index waits `wait_s` for that run. It then answers `state: indexing`: another
|
|
264
|
+
run is building the graph, and this call did nothing. `code_graph` never
|
|
265
|
+
waits. It answers from what is indexed so far, with `indexing: true` in its
|
|
266
|
+
value.
|
|
267
|
+
- **Progress is for a harness that starts the command itself.** A standard UTCP
|
|
268
|
+
client merges stderr into stdout, so no template of the manual carries
|
|
269
|
+
`--progress`. A harness that wants indexing to run in the background runs
|
|
270
|
+
`ph-code-graph call code_index --progress` in the tree and reads stdout and
|
|
271
|
+
stderr separately. Its model can call `code_graph` during the run.
|
|
272
|
+
- **The instructions for the model ship with the package**, as
|
|
273
|
+
`ph_code_graph/skills/code-graph/SKILL.md`: index first, ask the question that
|
|
274
|
+
you have, then read the narrow result.
|
|
275
|
+
|
|
276
|
+
The `ph-utcp` row of phern is one such harness. The next section describes what
|
|
277
|
+
it adds.
|
|
278
|
+
|
|
279
|
+
### Where the tools run, and what limits them
|
|
280
|
+
|
|
281
|
+
The tools run in the process of the `ph-code-graph` command. In a pH session,
|
|
282
|
+
`ctx.shell` starts that command for the agent that called the tool, in the
|
|
283
|
+
workspace of that agent and under its sandbox. A child agent in its own worktree
|
|
284
|
+
therefore indexes that worktree, and the harness process never loads
|
|
285
|
+
tree-sitter.
|
|
286
|
+
|
|
287
|
+
**`permissions-fs` does not screen the reads of these tools.** Its rules govern
|
|
288
|
+
calls through the `ctx.fs` of a session, and a command is not such a call. The
|
|
289
|
+
sandbox limits the command, as it limits `bash`.
|
|
290
|
+
|
|
291
|
+
**A read-only session protects the repository, not the index.** These tools
|
|
292
|
+
change no file in the tree. They build data that is used to query the tree. The
|
|
293
|
+
index is therefore kept in the scratch directory of the calling agent,
|
|
294
|
+
`$PH_HOME/scratch/<session>/<agent>/code-graph/`. That directory is writable in
|
|
295
|
+
every sandbox mode, and the tree is writable only in `workspace-write`. An agent
|
|
296
|
+
that may not write a tree can still index it, and cannot write the tree by doing
|
|
297
|
+
so. Every agent shares the grammar cache and the log of the command, under
|
|
298
|
+
`$PH_CACHE`, and the row grants those two paths.
|
|
299
|
+
|
|
300
|
+
## The two tools
|
|
301
|
+
|
|
302
|
+
**`code_index(paths?, glob?, exclude?, forget?)`** parses each file and records
|
|
303
|
+
what it defines and references.
|
|
304
|
+
|
|
305
|
+
When a call does not say what to index, the `index` block of the row decides:
|
|
306
|
+
|
|
307
|
+
```yaml
|
|
308
|
+
- id: code-graph
|
|
309
|
+
name: code-graph
|
|
310
|
+
config:
|
|
311
|
+
index:
|
|
312
|
+
paths: [packages/ph-core] # directories or files; default ["."]
|
|
313
|
+
include: ["**/*.py"] # default ["**/*"]; the language filter is the real filter
|
|
314
|
+
exclude: ["**/tests/**", "**/vendor/**"]
|
|
315
|
+
on_mount: true # index this when a root session starts; default true
|
|
316
|
+
languages: [python]
|
|
317
|
+
wait_s: 15 # how long code_index waits before it answers "indexing"
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
**Indexing starts without a request, and it does not delay a call.** Before the
|
|
321
|
+
agent of a root session takes its first step, `ph-utcp` starts one `code_index`
|
|
322
|
+
run over the `index` scope, in the background. The run is a job of that agent,
|
|
323
|
+
under its sandbox. The footer shows the count as it changes. The agent is told
|
|
324
|
+
when the run ends, if the run changed the graph. `on_mount: false` leaves
|
|
325
|
+
indexing to `code_index` calls.
|
|
326
|
+
|
|
327
|
+
A `code_index` call starts a run, or joins the same request if it is already in
|
|
328
|
+
progress, and waits `wait_s` for it:
|
|
329
|
+
|
|
330
|
+
- If the run ends in time, its result is the answer to the call.
|
|
331
|
+
- If not, the call answers `state: indexing`, with a line that gives the
|
|
332
|
+
progress of the run. The run continues, and the agent is told when it ends.
|
|
333
|
+
|
|
334
|
+
**One run at a time for each index.** A run holds the lock of the index while it
|
|
335
|
+
works. `state: indexing` means that another run holds the index and this call
|
|
336
|
+
did nothing. The totals beside it are what the graph holds so far. A different
|
|
337
|
+
request that is made during a run is not queued, so call `code_index` again
|
|
338
|
+
after the run ends. `code_graph` takes no lock and never waits. It answers from
|
|
339
|
+
what is indexed so far, and the `indexing` field of its value is `true` while a
|
|
340
|
+
run holds the index.
|
|
341
|
+
|
|
342
|
+
**A child agent builds its own graph only when it has a tree that it writes to
|
|
343
|
+
permanently.** A child that works in the checkout of its parent, or that can
|
|
344
|
+
only read the project, asks the graph of its parent. That includes every child
|
|
345
|
+
of a read-only fan-out. The results say which tree their paths belong to. A
|
|
346
|
+
child with its own worktree or overlay has files that diverge, so it gets its
|
|
347
|
+
own graph at its first call.
|
|
348
|
+
|
|
349
|
+
How the scope settings work:
|
|
350
|
+
|
|
351
|
+
- **`paths`** points the row at one package of a monorepo. A bare `code_index`
|
|
352
|
+
then stays under `max_files` and does not walk everything. A file that is
|
|
353
|
+
named directly in `paths` is indexed whatever is excluded.
|
|
354
|
+
- **`exclude`** is read against the workspace root, so `**/tests/**` means the
|
|
355
|
+
same thing under every entry of `paths`. A directory that it covers is never
|
|
356
|
+
entered.
|
|
357
|
+
- **The `glob` of a call** replaces `include` for that call.
|
|
358
|
+
- **The `exclude` of a call** is *added* to the `exclude` of the row. A call can
|
|
359
|
+
narrow the scope of a deployment, but it cannot remove one of its exclusions.
|
|
360
|
+
|
|
361
|
+
**Nothing watches the filesystem.** There is no daemon and no file watcher. The
|
|
362
|
+
index changes when `code_index` runs, and at no other time. After your own
|
|
363
|
+
edits, run it again. That is cheap by design, so it can be a habit.
|
|
364
|
+
|
|
365
|
+
**Indexing is incremental in two layers**, and the order matters:
|
|
366
|
+
|
|
367
|
+
1. *Does this file need to be opened?* The answer comes from **git or jj**,
|
|
368
|
+
which already hold a content-addressed tree (`ph.seams.changes`). A file that
|
|
369
|
+
the version control vouches for is never read. Measured on
|
|
370
|
+
`packages/ph-core/src/ph`: a re-index of 137 unchanged files went from 0.10 s
|
|
371
|
+
to **0.03 s**, with zero reads and zero hashes. The `tree_state` call cost
|
|
372
|
+
7 ms for the whole repository of 509 files.
|
|
373
|
+
2. *Is the opened content different?* The answer comes from the **sha256** of
|
|
374
|
+
the file, exactly as before. It never comes from an mtime. A checkout or a
|
|
375
|
+
rebase changes timestamps without changing content, and nobody attributes the
|
|
376
|
+
cost correctly when files are indexed again because git touched them.
|
|
377
|
+
|
|
378
|
+
Layer 1 decides only what to open, so the guarantee of layer 2 is unchanged. A
|
|
379
|
+
skip is always justified by a content hash and never by a clock. A tree with no
|
|
380
|
+
version control loses layer 1 and nothing else.
|
|
381
|
+
|
|
382
|
+
**`code_graph(mode, query?, …)`** answers six questions:
|
|
383
|
+
|
|
384
|
+
| `mode` | Question | What comes back |
|
|
385
|
+
|---|---|---|
|
|
386
|
+
| `search` | "something about workspace tiers" | Symbols ranked by FTS5 over name and docstring |
|
|
387
|
+
| `define` | "where exactly is `claim_slot`" | Every definition of that name |
|
|
388
|
+
| `callers` | "what calls this" | The calling symbol **and the calling line** |
|
|
389
|
+
| `callees` | "what does this call" | The definitions that it reaches |
|
|
390
|
+
| `impact` | "what breaks if I change this" | Transitive callers, ring by ring |
|
|
391
|
+
| `entities` | "what is big here" | Definitions by span, biggest first |
|
|
392
|
+
|
|
393
|
+
Every result carries `path:start-line-end-line`. That is the purpose of the
|
|
394
|
+
package: the model finds what matters in one call, and then reads exactly that.
|
|
395
|
+
|
|
396
|
+
## Using it from the RLM
|
|
397
|
+
|
|
398
|
+
There is nothing to do. In Code Mode the model gets one callable and a generated
|
|
399
|
+
SDK listing. *Every* registered tool is in that listing, including a tool that
|
|
400
|
+
arrives through a manual. These tools therefore arrive as
|
|
401
|
+
`await tools.code_index(...)` and `await tools.code_graph(...)`, with no Code
|
|
402
|
+
Mode work in this package. The `rlm-indexed` profile is `rlm-stable` plus this
|
|
403
|
+
bundle and the bundle of `ph-text-index`:
|
|
404
|
+
|
|
405
|
+
```bash
|
|
406
|
+
phern --profile rlm-indexed --provider llama --model <model> --mode tui
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
This package registers a `ph.bundles` entry point. That lets `phern` compose
|
|
410
|
+
the profile without importing this package. It also means that an install
|
|
411
|
+
without this distribution sees no `rlm-indexed`, and does not see one that
|
|
412
|
+
fails at mount.
|
|
413
|
+
|
|
414
|
+
## It is a name-based graph, and it says so
|
|
415
|
+
|
|
416
|
+
A reference records the *name* that it used, and `callers` and `callees` join on
|
|
417
|
+
that name. Two `register` methods in two classes are one name to this index.
|
|
418
|
+
|
|
419
|
+
Every answer that could be ambiguous carries `definitions`, the number of places
|
|
420
|
+
that define the name. When that number is above one, the rendered text says so
|
|
421
|
+
directly. The model then sees the ambiguity, and is not given one of the
|
|
422
|
+
possibilities without notice. `mode=define` lists the definitions. It does not
|
|
423
|
+
say which definition an edge belongs to. No name-based index can say that.
|
|
424
|
+
|
|
425
|
+
A resolved graph needs an import graph, scope, and type inference for each
|
|
426
|
+
language. In the Rust/Node CodeGraph that is 29,708 lines of TypeScript, and
|
|
427
|
+
this package deliberately did not port it. Name-based matching answers most of
|
|
428
|
+
what an agent asks, and reports where it cannot. That is a better trade than a
|
|
429
|
+
resolved graph for one language.
|
|
430
|
+
|
|
431
|
+
## Why Python-native, and not a wrapper
|
|
432
|
+
|
|
433
|
+
The obvious plan was to add the Rust/Node CodeGraph as a submodule and wrap its
|
|
434
|
+
kernel with maturin. The investigation rejected that plan, for three reasons in
|
|
435
|
+
increasing order of importance:
|
|
436
|
+
|
|
437
|
+
1. **`codegraph-kernel` is a `#[napi]` crate**: `crate-type = ["cdylib"]`
|
|
438
|
+
against `napi` and `napi-derive`. maturin builds PyO3 and cffi extensions. A
|
|
439
|
+
wrapper would need a fork of the crate with a rewritten boundary.
|
|
440
|
+
2. **It is only an extractor.** Its whole export surface is `extract_file`,
|
|
441
|
+
`contract_info`, `grammar_info`, and two `cfnptr` helpers. In its own words:
|
|
442
|
+
"tree-sitter parse+extract with one JS boundary crossing per file". It also
|
|
443
|
+
has a complete TS/WASM fallback, so it is an accelerator and not the core.
|
|
444
|
+
3. **The intelligence is TypeScript**: `resolution/` has 29,708 lines,
|
|
445
|
+
`extraction/` 25,612, `mcp/` 13,940, `graph/` 6,023, and `db/` 5,475. That is
|
|
446
|
+
112,612 lines against 25,242 in the kernel. A wrapper around the kernel gives
|
|
447
|
+
a parser and leaves the product behind.
|
|
448
|
+
|
|
449
|
+
Python already has the parser. `tree-sitter-language-pack` is a 3.7 MB wheel. It
|
|
450
|
+
calls 26 languages bundled (python, ts/tsx/js, rust, go, java, csharp, c, cpp,
|
|
451
|
+
ruby, php, swift, kotlin, scala, dart, lua, r, and more) and has 371 available.
|
|
452
|
+
**None of them works offline on a fresh cache.** Measured 2026-11-11 with 1.16.2
|
|
453
|
+
and 1.25.0: with no network and an empty cache, no parser can be obtained,
|
|
454
|
+
including the python parser. With the network, the bundled set arrives as one
|
|
455
|
+
archive from GitHub the first time a parser is requested. After that, the pack
|
|
456
|
+
needs only the cache.
|
|
457
|
+
|
|
458
|
+
## Why `sqlite3` and not `pyturso`
|
|
459
|
+
|
|
460
|
+
The session log of pH runs on turso, so this package is the exception, and the
|
|
461
|
+
reason is measured. Against a real 40 MB CodeGraph database (11,396 nodes,
|
|
462
|
+
36,830 edges), `pyturso` served indexed lookups, joins, and aggregates
|
|
463
|
+
correctly. It then failed in three ways:
|
|
464
|
+
|
|
465
|
+
- **FTS5 is absent.** `pyturso` does not see a `CREATE VIRTUAL TABLE … USING
|
|
466
|
+
fts5`, or its shadow tables. `search` needs FTS5.
|
|
467
|
+
- **`Recursive CTEs are not yet supported`.** `impact` needs a recursive CTE.
|
|
468
|
+
- **`json_extract` silently returns NULL** where SQLite returns the value:
|
|
469
|
+
`0.95` against `None`, with the same rows and the same expression. A wrong
|
|
470
|
+
answer with no error is worse than a missing feature.
|
|
471
|
+
|
|
472
|
+
Two of the six modes here are exactly the two that turso cannot serve. Stdlib
|
|
473
|
+
`sqlite3` (SQLite 3.45, with FTS5 compiled in) adds no dependency and serves all
|
|
474
|
+
six. If turso gets both features, `_store.py` changes one import.
|
|
475
|
+
|
|
476
|
+
## Extraction: two passes, and why
|
|
477
|
+
|
|
478
|
+
- **`process()`** is the intelligence layer of the pack. It gives definitions
|
|
479
|
+
with kinds, the imports of the file, docstrings, and comments. It does not
|
|
480
|
+
give *references*: `SymbolInfo` says that a function exists, never who calls
|
|
481
|
+
it.
|
|
482
|
+
- **The tags query** is the `tags.scm` of tree-sitter for each language. GitHub
|
|
483
|
+
builds its code navigation on the same data. It gives `@definition.*` and,
|
|
484
|
+
most important, `@reference.call`.
|
|
485
|
+
|
|
486
|
+
Measured over `packages/ph-core/src/ph`: 136 files and 1.58 MiB take 219 ms
|
|
487
|
+
through `process()` and 108 ms through the tags query. An index of the core of
|
|
488
|
+
this repository takes 2.0 s end to end, including the SQLite writes, and 0.1 s
|
|
489
|
+
when nothing has changed. The second parse gives more value than it costs.
|
|
490
|
+
|
|
491
|
+
Four details are easy to get wrong. Each is handled one time in `_extract`, and
|
|
492
|
+
each has a test:
|
|
493
|
+
|
|
494
|
+
- **`ProcessConfig` spans are 0-based.** Tree-sitter points are also 0-based,
|
|
495
|
+
and every line that pH shows a model is 1-based. The numbers are normalized on
|
|
496
|
+
the way out, so nothing downstream has to remember which pass a number came
|
|
497
|
+
from. An error here gives a pointer that is off by one, which makes a model
|
|
498
|
+
stop trusting a tool.
|
|
499
|
+
- **`SymbolInfo.doc` is always `None`.** The pack does not set it (measured
|
|
500
|
+
against a file full of docstrings). Code that trusted it would have shipped an
|
|
501
|
+
index with an empty `doc` column and a `search` mode that matched names only.
|
|
502
|
+
The prose is on two other channels. `result.docstrings` carries the pack's own
|
|
503
|
+
`associated_item`. `result.comments` carries doc comments (Rust `///`,
|
|
504
|
+
TypeScript block comments, Go `//`). Those comments have no association, so
|
|
505
|
+
the rule is positional and narrow: the last line of the comment must be
|
|
506
|
+
*immediately* above the first line of the definition.
|
|
507
|
+
- **The `span.end_line` of a comment node can be one past the line that the
|
|
508
|
+
comment occupies**, because the `///` node of Rust includes its trailing
|
|
509
|
+
newline. With `end_line`, the rule missed the definition directly below *and*
|
|
510
|
+
matched one a blank line away. The line count now comes from the text of the
|
|
511
|
+
comment.
|
|
512
|
+
- **The tags query of TypeScript is only part of the query.** Its grammar
|
|
513
|
+
extends the javascript grammar, and upstream splits the queries in the same
|
|
514
|
+
way. `class`, `function`, and `call` are in the *javascript* query. A `.ts`
|
|
515
|
+
file that is matched against the typescript query alone yields two tags and no
|
|
516
|
+
calls. `INHERITS` fixes that.
|
|
517
|
+
- **The python query of the pack missed module-level constants.** It looks for
|
|
518
|
+
an assignment inside an `expression_statement`, and the grammar of the pack
|
|
519
|
+
puts an assignment directly under `module`. `SUPPLEMENTS` adds a pattern that
|
|
520
|
+
fits the tree.
|
|
521
|
+
|
|
522
|
+
## Limits for each language
|
|
523
|
+
|
|
524
|
+
Tests cover both of these, so they are known shapes and not surprises:
|
|
525
|
+
|
|
526
|
+
- **C has definitions but no call references.** Its tags query ships no
|
|
527
|
+
`@reference.*` captures. `callers` and `callees` are therefore empty for C,
|
|
528
|
+
while `search`, `define`, and `entities` work.
|
|
529
|
+
- **A bare Ruby send is not seen.** `helper` without parentheses parses as an
|
|
530
|
+
identifier and not as a `call`, so only `helper()` becomes an edge.
|
|
531
|
+
|
|
532
|
+
Only `reference.call` is stored. The tags queries also emit `reference.type`,
|
|
533
|
+
`reference.class`, and more. Each is a real relationship, but with them
|
|
534
|
+
`callers` would return every place that only *mentions* a type.
|
|
535
|
+
|
|
536
|
+
## Where things live
|
|
537
|
+
|
|
538
|
+
- **The index, in a session**: `graph.db` in the scratch directory of the agent
|
|
539
|
+
that owns the tree, `$PH_HOME/scratch/<session>/<agent>/code-graph/`. The
|
|
540
|
+
owner is the calling agent, or the ancestor whose tree it shares. A child in
|
|
541
|
+
its own worktree indexes its own tree, so two agents on different trees never
|
|
542
|
+
answer from the line numbers of the other. A resumed session has the same
|
|
543
|
+
session id and agent id, and therefore the same index. `code_index` then
|
|
544
|
+
parses again only what changed. The cost is that a new session builds its own
|
|
545
|
+
index and does not find the index of the last session.
|
|
546
|
+
- **The index, from a shell**: `--state DIR` puts it in `DIR`, and so does
|
|
547
|
+
`$PH_TOOL_STATE`. `ph-utcp` sets that variable for each call, and another
|
|
548
|
+
harness can set it. With neither, the index is
|
|
549
|
+
`$PH_CACHE/code-graph/<digest of the workspace root>/graph.db`. It is under
|
|
550
|
+
the cache root because it can be rebuilt from the source. It is keyed by the
|
|
551
|
+
root so that two checkouts do not answer the questions of each other. That
|
|
552
|
+
directory records its workspace. When the workspace has been gone for a week,
|
|
553
|
+
the next index removes the directory.
|
|
554
|
+
- **The grammars**: `$PH_CACHE/tree-sitter`. The engine row sets this when the
|
|
555
|
+
command mounts it.
|
|
556
|
+
- **The writer lock**: `graph.db.lock` beside the database. It allows one
|
|
557
|
+
indexing run at a time for each index, across every process on the workspace.
|
|
558
|
+
A second run waits `wait_s` for the first. It then finds most of the tree
|
|
559
|
+
current, or answers `state: indexing`.
|
|
560
|
+
- **The log of the command**: `$PH_CACHE/logs/ph-code-graph.log`. A call prints
|
|
561
|
+
one JSON object, and prints nothing on stderr except the progress lines that
|
|
562
|
+
`--progress` asks for. A traceback therefore goes to the log.
|
|
563
|
+
|
|
564
|
+
The location of the grammars is important. The pack keeps every grammar that it
|
|
565
|
+
uses in a writable cache. That includes its bundled grammars, which it downloads
|
|
566
|
+
there on first use. The pack **fails hard** when it cannot create that
|
|
567
|
+
directory. Its own default is `$XDG_CACHE_HOME/…`, which is correct on a laptop
|
|
568
|
+
and wrong in a container with a read-only `HOME` (both measured). The row
|
|
569
|
+
therefore gives the pack the root that pH already uses for rebuildable
|
|
570
|
+
artifacts. When even that root is not writable, the row refuses to mount with
|
|
571
|
+
one sentence, not with a `pathlib` traceback. From the command, that sentence is
|
|
572
|
+
the answer of the call, with the code `MOUNT_REFUSED`.
|
|
573
|
+
|
|
574
|
+
`TREE_SITTER_LANGUAGE_PACK_CACHE_DIR` still wins if an operator set it. Their
|
|
575
|
+
value is what keeps the variable meaningful.
|
|
576
|
+
|
|
577
|
+
Expect two consequences:
|
|
578
|
+
|
|
579
|
+
- The **first** index on a cold grammar cache pays for the grammar fetch.
|
|
580
|
+
Measured on `packages/ph-core/src/ph`: 9 s cold against 2 s warm, and 0.1 s
|
|
581
|
+
when nothing changed.
|
|
582
|
+
- A command that cannot reach GitHub indexes nothing until the cache is filled.
|
|
583
|
+
That applies to the command of a confined agent, and to a machine with no
|
|
584
|
+
network. Each file is skipped, with a reason that names `/code-graph install`.
|
|
585
|
+
|
|
586
|
+
Provision the grammars deliberately, not during the turn of a user:
|
|
587
|
+
|
|
588
|
+
```
|
|
589
|
+
/code-graph status # how many grammars are ready, and where they are
|
|
590
|
+
/code-graph install # load each one now, and name any that will not load
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
This is a **command**, not a tool. A person asks the harness to provision, and
|
|
594
|
+
that costs no model turn. In a session, it runs `ph-code-graph status` or
|
|
595
|
+
`ph-code-graph install` and shows the output. From a shell, type those commands
|
|
596
|
+
directly. `phern doctor` reports the path of the index, whether the index is
|
|
597
|
+
built, and whether the manual was discovered. It does not start the command.
|
|
598
|
+
`install` fetches all eighteen grammars in about three seconds. It runs outside
|
|
599
|
+
the sandbox, because it needs the network.
|
|
600
|
+
|
|
601
|
+
## The skill
|
|
602
|
+
|
|
603
|
+
The package installs a `code-graph` skill. An RLM under progressive disclosure
|
|
604
|
+
sees one line in its catalog and can read the page when it needs it. The page
|
|
605
|
+
says: index first, ask the question that you have, then `read` the narrow
|
|
606
|
+
result. It also says that a name-based graph reports ambiguity and does not
|
|
607
|
+
resolve it.
|
|
608
|
+
|
|
609
|
+
The row registers the skill. A directory scan does not find it. The skill
|
|
610
|
+
therefore arrives exactly when the tools do, and is removed with them.
|
|
611
|
+
`skills-progressive` ships an empty `paths` on purpose, because a distribution
|
|
612
|
+
installs a skill deliberately (I7).
|
|
613
|
+
|
|
614
|
+
## Tests
|
|
615
|
+
|
|
616
|
+
`tests/test_code_graph.py` runs the real parser and the real SQLite index. It
|
|
617
|
+
deliberately tests three things:
|
|
618
|
+
|
|
619
|
+
- **Line spans against the file.** Each span is checked by slicing the real
|
|
620
|
+
source at the span, not against an expected number. A pointer that is off by
|
|
621
|
+
one still looks plausible in a diff.
|
|
622
|
+
- **The enclosing-symbol rule.** A call inside a nested closure belongs to the
|
|
623
|
+
closure, not to the 40-line function around it.
|
|
624
|
+
- **Incrementality.** A second index pass over an unchanged tree parses nothing,
|
|
625
|
+
and a file that was touched but not modified stays unchanged.
|
|
626
|
+
|
|
627
|
+
`tests/test_code_graph_cli.py` starts real processes, because its subject exists
|
|
628
|
+
only between processes:
|
|
629
|
+
|
|
630
|
+
- The command answers with the values that the row answers with.
|
|
631
|
+
- Two processes that index one tree report counts that add up.
|
|
632
|
+
- A real UTCP client registers the manual and calls a tool, with a list argument
|
|
633
|
+
and with optional arguments left out.
|
|
634
|
+
|
|
635
|
+
`tests/test_code_graph_row.py` covers the row that a session mounts:
|
|
636
|
+
|
|
637
|
+
- The tools keep their names, values, rendered text, and error codes.
|
|
638
|
+
- An agent in its own tree indexes that tree.
|
|
639
|
+
- A confined agent that may not write its tree still indexes it.
|
|
640
|
+
- A fresh interpreter that mounts only the rows of the session never loads
|
|
641
|
+
tree-sitter.
|