slurp-graph 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- slurp_graph-0.1.0/.gitignore +14 -0
- slurp_graph-0.1.0/.python-version +1 -0
- slurp_graph-0.1.0/PKG-INFO +356 -0
- slurp_graph-0.1.0/README.md +333 -0
- slurp_graph-0.1.0/images/SlurpLogo.png +0 -0
- slurp_graph-0.1.0/images/SlurpLogo_SinFondo.png +0 -0
- slurp_graph-0.1.0/pyproject.toml +50 -0
- slurp_graph-0.1.0/slurp/__init__.py +3 -0
- slurp_graph-0.1.0/slurp/audit.py +85 -0
- slurp_graph-0.1.0/slurp/benchmark.py +224 -0
- slurp_graph-0.1.0/slurp/budget.py +154 -0
- slurp_graph-0.1.0/slurp/cli.py +487 -0
- slurp_graph-0.1.0/slurp/diff.py +336 -0
- slurp_graph-0.1.0/slurp/exporter.py +79 -0
- slurp_graph-0.1.0/slurp/formatter.py +263 -0
- slurp_graph-0.1.0/slurp/ignore.py +91 -0
- slurp_graph-0.1.0/slurp/injector.py +179 -0
- slurp_graph-0.1.0/slurp/loader.py +434 -0
- slurp_graph-0.1.0/slurp/mcp.py +195 -0
- slurp_graph-0.1.0/slurp/scorer.py +322 -0
- slurp_graph-0.1.0/slurp/viz.py +339 -0
- slurp_graph-0.1.0/tests/__init__.py +0 -0
- slurp_graph-0.1.0/tests/conftest.py +199 -0
- slurp_graph-0.1.0/tests/test_audit.py +406 -0
- slurp_graph-0.1.0/tests/test_benchmark.py +501 -0
- slurp_graph-0.1.0/tests/test_budget.py +465 -0
- slurp_graph-0.1.0/tests/test_cli.py +1130 -0
- slurp_graph-0.1.0/tests/test_diff.py +550 -0
- slurp_graph-0.1.0/tests/test_exporter.py +256 -0
- slurp_graph-0.1.0/tests/test_formatter.py +524 -0
- slurp_graph-0.1.0/tests/test_ignore.py +259 -0
- slurp_graph-0.1.0/tests/test_injector.py +451 -0
- slurp_graph-0.1.0/tests/test_loader.py +767 -0
- slurp_graph-0.1.0/tests/test_mcp.py +471 -0
- slurp_graph-0.1.0/tests/test_scorer.py +644 -0
- slurp_graph-0.1.0/uv.lock +807 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.12
|
|
@@ -0,0 +1,356 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: slurp-graph
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Token-budget-aware graph navigation for AI coding agents. Serve exactly the noodles your LLM needs.
|
|
5
|
+
Project-URL: Homepage, https://github.com/CarlosVallejoRuiz/slurp
|
|
6
|
+
Project-URL: Repository, https://github.com/CarlosVallejoRuiz/slurp
|
|
7
|
+
Author-email: Juan Carlos Vallejo Ruiz <juancarlos@prismastats.es>
|
|
8
|
+
License: MIT
|
|
9
|
+
Keywords: ai-tools,cli,graphify,knowledge-graph,llm,token-optimization
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
13
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
14
|
+
Classifier: Topic :: Software Development :: Documentation
|
|
15
|
+
Requires-Python: >=3.12
|
|
16
|
+
Requires-Dist: anthropic>=0.106.0
|
|
17
|
+
Requires-Dist: click>=8.4.1
|
|
18
|
+
Requires-Dist: networkx>=3.6.1
|
|
19
|
+
Requires-Dist: pathspec>=1.1.1
|
|
20
|
+
Requires-Dist: rich>=15.0.0
|
|
21
|
+
Requires-Dist: tiktoken>=0.13.0
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
|
|
24
|
+
# slurp
|
|
25
|
+
|
|
26
|
+

|
|
27
|
+

|
|
28
|
+

|
|
29
|
+

|
|
30
|
+
|
|
31
|
+
> *graphify builds the bowl. slurp serves exactly the noodles your LLM needs.*
|
|
32
|
+
|
|
33
|
+
A knowledge graph is a bowl of ramen — thousands of nodes tangled together. Your LLM doesn't need the whole bowl. Slurp scores every node against your query, then greedily selects the highest-relevance subgraph that fits within your token budget — and tells you exactly what it picked and why.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Benchmark
|
|
38
|
+
|
|
39
|
+
Tested on a real **PrismaStats** codebase: 2,111 nodes, 28,412 tokens total.
|
|
40
|
+
|
|
41
|
+
| Query | Budget 2k | Budget 4k | Budget 8k |
|
|
42
|
+
|---|---|---|---|
|
|
43
|
+
| `"auth flow"` | **97.1%** saved | 96.3% saved | 95.2% saved |
|
|
44
|
+
| `"prisma schema"` | 95.8% saved | 94.2% saved | 93.8% saved |
|
|
45
|
+
| `"database pool"` | 93.1% saved | 89.1% saved | 85.1% saved |
|
|
46
|
+
|
|
47
|
+
**Mean savings: 93.3% · p50: 94.2% · Best case: 97.1%**
|
|
48
|
+
|
|
49
|
+
Even the worst case — `"database pool"` at budget 8k — injects 85% fewer tokens than the full graph.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Install
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
pip install slurp-graph
|
|
57
|
+
|
|
58
|
+
# or with uv
|
|
59
|
+
uv add slurp-graph
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
> PyPI package: `slurp-graph` — CLI command: `slurp`
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Quickstart
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
slurp "auth flow" --graph graph.json --budget 4000
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
╭─ Slurp — Subgraph for: "auth flow" (budget: 4,000 tokens) ──────────────╮
|
|
74
|
+
│ Selected 5/2111 nodes · 847/4,000 tokens used (21.2%) │
|
|
75
|
+
╰───────────────────────────────────────────────────────────────────────────╯
|
|
76
|
+
|
|
77
|
+
## Relevant Nodes
|
|
78
|
+
|
|
79
|
+
### authenticate_user (function) · score: 0.94
|
|
80
|
+
Validates user credentials and returns JWT token.
|
|
81
|
+
→ File: src/auth/service.py
|
|
82
|
+
|
|
83
|
+
### JWTMiddleware (class) · score: 0.87
|
|
84
|
+
Intercepts HTTP requests and validates Authorization header.
|
|
85
|
+
→ File: src/middleware/jwt.py
|
|
86
|
+
|
|
87
|
+
### hash_password (function) · score: 0.71
|
|
88
|
+
Hashes password using bcrypt with a cost factor of 12.
|
|
89
|
+
→ File: src/auth/utils.py
|
|
90
|
+
|
|
91
|
+
## Key Relationships
|
|
92
|
+
- JWTMiddleware → calls → authenticate_user
|
|
93
|
+
- authenticate_user → calls → hash_password
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
💡 2106 additional connected nodes available — increase --budget to include them
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Add `--inject-code` to embed the actual function body next to each node:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
slurp "auth flow" --graph graph.json --budget 4000 --inject-code
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
````
|
|
106
|
+
### authenticate_user (function) · score: 0.94
|
|
107
|
+
Validates user credentials and returns JWT token.
|
|
108
|
+
→ File: src/auth/service.py
|
|
109
|
+
|
|
110
|
+
```python
|
|
111
|
+
def authenticate_user(username: str, password: str) -> dict | None:
|
|
112
|
+
user = db.query(User).filter_by(username=username).first()
|
|
113
|
+
if not user or not bcrypt.checkpw(password.encode(), user.password_hash):
|
|
114
|
+
return None
|
|
115
|
+
return {"token": jwt.encode({"sub": user.id}, SECRET_KEY)}
|
|
116
|
+
```
|
|
117
|
+
````
|
|
118
|
+
|
|
119
|
+
Pipe the output directly into your LLM prompt, save it to a file, or use `slurp export` to format it as a ready-to-paste system prompt block.
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## Commands
|
|
124
|
+
|
|
125
|
+
### `slurp QUERY`
|
|
126
|
+
|
|
127
|
+
The main command. Scores all graph nodes against your query and selects the optimal subgraph within the token budget.
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
slurp "auth flow" --graph graph.json --budget 4000
|
|
131
|
+
slurp "payment processing" --format json
|
|
132
|
+
slurp "JWT validation" --explain
|
|
133
|
+
slurp "database schema" --inject-code --min-score 0.3
|
|
134
|
+
slurp "prisma models" --backend openai
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
| Flag | Default | Description |
|
|
138
|
+
|---|---|---|
|
|
139
|
+
| `--graph`, `-g` | auto-discover | Path to `graph.json`. |
|
|
140
|
+
| `--budget`, `-b` | `4000` | Token budget for subgraph selection. |
|
|
141
|
+
| `--format`, `-f` | `markdown` | Output format: `markdown`, `json`, or `yaml`. |
|
|
142
|
+
| `--model`, `-m` | `cl100k_base` | Tiktoken encoding for token counting. |
|
|
143
|
+
| `--explain` | off | Print per-node score breakdown: final / structural / semantic. |
|
|
144
|
+
| `--no-audit` | off | Skip writing to `.slurp/audit.jsonl`. |
|
|
145
|
+
| `--neighbor-decay` | `0.7` | Score multiplier applied to neighbors of each selected node. |
|
|
146
|
+
| `--min-score` | `0.15` | Minimum relevance score; nodes below this are excluded before selection. |
|
|
147
|
+
| `--viz` | off | Open an interactive graph visualization in the browser. |
|
|
148
|
+
| `--ignore-file` | `.slurpignore` | Path to node exclusion rules. |
|
|
149
|
+
| `--backend` | `tfidf` | Scoring backend: `tfidf` (default), `openai`, or `anthropic`. |
|
|
150
|
+
| `--inject-code` | off | Embed source code blocks for each selected node (requires ≤30 nodes). |
|
|
151
|
+
| `--project-root` | graph dir | Root directory for resolving `source_file` paths. |
|
|
152
|
+
|
|
153
|
+
**Auto-discovery** (when `--graph` is omitted):
|
|
154
|
+
|
|
155
|
+
1. `./graph.json`
|
|
156
|
+
2. `./graphify-out/graph.json`
|
|
157
|
+
3. `./.graphify/graph.json`
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
### `slurp stats`
|
|
162
|
+
|
|
163
|
+
Print node and edge counts for a graph file.
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
slurp stats --graph graph.json
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
Graph: graph.json
|
|
171
|
+
Nodes: 2111
|
|
172
|
+
Edges: 4823
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
### `slurp audit`
|
|
178
|
+
|
|
179
|
+
Show the history of queries logged to `.slurp/audit.jsonl`, plus the most frequently selected nodes.
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
slurp audit
|
|
183
|
+
slurp audit --top-nodes 20
|
|
184
|
+
slurp audit --audit-dir /custom/.slurp
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Every query is appended as a JSON line (unless `--no-audit` is passed). Useful for tracking which parts of your codebase an AI agent visits most.
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
### `slurp diff`
|
|
192
|
+
|
|
193
|
+
Compare two graph versions and report the impact of changes.
|
|
194
|
+
|
|
195
|
+
```bash
|
|
196
|
+
slurp diff old.json new.json
|
|
197
|
+
slurp diff old.json new.json --hops 2 --viz
|
|
198
|
+
slurp diff old.json new.json --budget 4000
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Reports added/removed/modified nodes and edges, computes an impact score based on centrality, and optionally opens a diff-colored visualization (green=added, red=removed, yellow=modified, grey=unchanged). Pass `--budget` to further select the most relevant affected nodes.
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
### `slurp export`
|
|
206
|
+
|
|
207
|
+
Export a context block ready to paste into an AI system prompt.
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
slurp export "auth flow" --format claude # <context> XML tags
|
|
211
|
+
slurp export "auth flow" --format chatgpt # [CODEBASE CONTEXT] block
|
|
212
|
+
slurp export "auth flow" --format claudemd # ## Codebase Context for CLAUDE.md
|
|
213
|
+
slurp export "auth flow" --output context.md
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
All three formats include query, nodes selected/total, tokens used/budget, and coverage %.
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
### `slurp serve`
|
|
221
|
+
|
|
222
|
+
Start an MCP stdio server (JSON-RPC 2.0) that exposes the `slurp_query` tool.
|
|
223
|
+
|
|
224
|
+
```bash
|
|
225
|
+
slurp serve --graph graph.json
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
See [MCP Integration](#mcp-integration) for configuration.
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
### `slurp benchmark`
|
|
233
|
+
|
|
234
|
+
Measure real token savings across queries and budgets.
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
slurp benchmark \
|
|
238
|
+
--graph graph.json \
|
|
239
|
+
--queries "auth flow" --queries "schema validation" \
|
|
240
|
+
--budget 2000 --budget 4000 --budget 8000
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
Outputs a per-run table and aggregate stats: mean savings, p50/p90/p95, best/worst case, and precision (fraction of relevant nodes captured).
|
|
244
|
+
|
|
245
|
+
---
|
|
246
|
+
|
|
247
|
+
## Works with graphify
|
|
248
|
+
|
|
249
|
+
Slurp is the query layer for [`graphify`](https://github.com/CarlosVallejoRuiz/graphify). Run graphify on your codebase, point slurp at the output.
|
|
250
|
+
|
|
251
|
+
```bash
|
|
252
|
+
graphify . # generates graphify-out/graph.json
|
|
253
|
+
slurp "auth flow" --budget 4000 # auto-discovers graphify-out/graph.json
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
**Supported node fields:**
|
|
257
|
+
|
|
258
|
+
```json
|
|
259
|
+
{
|
|
260
|
+
"id": "authenticate_user",
|
|
261
|
+
"label": "authenticate_user",
|
|
262
|
+
"type": "function",
|
|
263
|
+
"description": "Validates credentials and returns JWT.",
|
|
264
|
+
"importance": 9,
|
|
265
|
+
"source_file": "src/auth/service.py",
|
|
266
|
+
"source_location": "L42"
|
|
267
|
+
}
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
The `type`, `description`, `importance`, `source_file`, and `source_location` fields are optional but improve scoring and enable `--inject-code`. Any graph with `id` + `label` on nodes and `source`/`target` on edges will work.
|
|
271
|
+
|
|
272
|
+
Both `links` (graphify/NetworkX serialization) and `edges` are supported. Additional formats are auto-detected by extension:
|
|
273
|
+
|
|
274
|
+
| Extension | Format |
|
|
275
|
+
|---|---|
|
|
276
|
+
| `.json` | graphify or generic JSON |
|
|
277
|
+
| `.graphml` | GraphML (NetworkX / yEd / Gephi) |
|
|
278
|
+
| `.csv` | Neo4j export (nodes CSV + sibling relationships CSV) |
|
|
279
|
+
|
|
280
|
+
Use `slurp convert` or the `convert_graph()` API to export between formats.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
## MCP Integration
|
|
285
|
+
|
|
286
|
+
Run slurp as an MCP server so Claude Code (or any MCP-compatible agent) can query the graph directly.
|
|
287
|
+
|
|
288
|
+
**`.mcp.json`:**
|
|
289
|
+
|
|
290
|
+
```json
|
|
291
|
+
{
|
|
292
|
+
"mcpServers": {
|
|
293
|
+
"slurp": {
|
|
294
|
+
"command": "/path/to/.venv/bin/slurp",
|
|
295
|
+
"args": ["serve", "--graph", "/path/to/graphify-out/graph.json"]
|
|
296
|
+
}
|
|
297
|
+
}
|
|
298
|
+
}
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
**Tool exposed:** `slurp_query(query: str, budget: int = 4000) → str`
|
|
302
|
+
|
|
303
|
+
Claude Code calls this automatically when it needs codebase context. The server runs over stdio and returns the formatted markdown subgraph — no HTTP, no ports.
|
|
304
|
+
|
|
305
|
+
---
|
|
306
|
+
|
|
307
|
+
## .slurpignore
|
|
308
|
+
|
|
309
|
+
Exclude nodes by type, file path, or ID pattern. Create `.slurpignore` in your project root:
|
|
310
|
+
|
|
311
|
+
```
|
|
312
|
+
# Exclude documentation nodes
|
|
313
|
+
type:document
|
|
314
|
+
type:markdown
|
|
315
|
+
|
|
316
|
+
# Exclude test files
|
|
317
|
+
file:tests/**
|
|
318
|
+
file:**/*.test.ts
|
|
319
|
+
|
|
320
|
+
# Exclude generated code
|
|
321
|
+
id:generated_*
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
Pass a custom path with `--ignore-file path/to/.slurpignore`.
|
|
325
|
+
|
|
326
|
+
---
|
|
327
|
+
|
|
328
|
+
## Design decisions
|
|
329
|
+
|
|
330
|
+
**Power-iteration PageRank without numpy.** `nx.pagerank()` requires numpy. Slurp implements a 20-line pure-Python power-iteration algorithm (convergence: `Σ|rank_new − rank_old| < N × tol`). Same result, no heavy dependency.
|
|
331
|
+
|
|
332
|
+
**TF-IDF without scikit-learn.** Hand-rolled TF-IDF with smoothed IDF (`log((N+1)/(df+1)) + 1`) and cosine similarity. The tokenizer splits camelCase and snake_case, so `authenticate_user` scores on both `authenticate` and `user`. The `score_nodes()` interface is backend-agnostic — swap to real embeddings with `--backend openai` or `--backend anthropic` without touching any caller.
|
|
333
|
+
|
|
334
|
+
**YAML serializer without PyYAML.** `_yaml_scalar()` renders Python primitives as valid YAML scalars using `json.dumps()` for strings that need quoting (JSON string literals are valid YAML 1.1). No PyYAML dependency.
|
|
335
|
+
|
|
336
|
+
**`lru_cache` on the tiktoken encoder.** `tiktoken.get_encoding()` reads tokenizer data from disk on first call. Caching with `lru_cache(maxsize=8)` means repeated token-counting calls within a single run hit memory, not disk.
|
|
337
|
+
|
|
338
|
+
**`+0.3` score boost for `file_type == "code"` nodes (clamped to 1.0).** Documentation nodes compete unfairly with code in technical queries. The boost is bounded so it cannot override a genuinely high structural+semantic score.
|
|
339
|
+
|
|
340
|
+
**`--inject-code` capped at 30 nodes.** Code blocks are 50–200 tokens each. At 30 nodes, that's up to 6,000 extra tokens — manageable. At 200 nodes it would explode the context budget. The cap is enforced in both the CLI (warning message) and `inject_code()` (hard guard), so the formatter never receives oversized input.
|
|
341
|
+
|
|
342
|
+
---
|
|
343
|
+
|
|
344
|
+
## Roadmap
|
|
345
|
+
|
|
346
|
+
- ✅ **v0.1.0** — `loader`, `scorer`, `budget`, `formatter`, `audit` — core pipeline, full tests, `slurp QUERY` + `slurp stats`
|
|
347
|
+
- ✅ **v0.2.0** — `--explain`, `.slurpignore`, `--viz` interactive HTML, `--min-score`, camelCase/snake_case tokenizer, `--neighbor-decay`
|
|
348
|
+
- ✅ **v0.3.0** — `slurp serve` (MCP stdio), `slurp diff`, `slurp export` (claude/chatgpt/claudemd), PyPI publish as `slurp-graph`
|
|
349
|
+
- ✅ **v0.4.0** — `--backend openai|anthropic` (optional embeddings), `slurp benchmark`, GraphML + Neo4j CSV loader, `convert_graph()`
|
|
350
|
+
- ✅ **v0.5.0** — `--inject-code`: extract real function bodies from source files and embed them in the context output
|
|
351
|
+
|
|
352
|
+
---
|
|
353
|
+
|
|
354
|
+
## License
|
|
355
|
+
|
|
356
|
+
MIT © [Juan Carlos Vallejo Ruiz](https://github.com/CarlosVallejoRuiz)
|
|
@@ -0,0 +1,333 @@
|
|
|
1
|
+
# slurp
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
> *graphify builds the bowl. slurp serves exactly the noodles your LLM needs.*
|
|
9
|
+
|
|
10
|
+
A knowledge graph is a bowl of ramen — thousands of nodes tangled together. Your LLM doesn't need the whole bowl. Slurp scores every node against your query, then greedily selects the highest-relevance subgraph that fits within your token budget — and tells you exactly what it picked and why.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Benchmark
|
|
15
|
+
|
|
16
|
+
Tested on a real **PrismaStats** codebase: 2,111 nodes, 28,412 tokens total.
|
|
17
|
+
|
|
18
|
+
| Query | Budget 2k | Budget 4k | Budget 8k |
|
|
19
|
+
|---|---|---|---|
|
|
20
|
+
| `"auth flow"` | **97.1%** saved | 96.3% saved | 95.2% saved |
|
|
21
|
+
| `"prisma schema"` | 95.8% saved | 94.2% saved | 93.8% saved |
|
|
22
|
+
| `"database pool"` | 93.1% saved | 89.1% saved | 85.1% saved |
|
|
23
|
+
|
|
24
|
+
**Mean savings: 93.3% · p50: 94.2% · Best case: 97.1%**
|
|
25
|
+
|
|
26
|
+
Even the worst case — `"database pool"` at budget 8k — injects 85% fewer tokens than the full graph.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## Install
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
pip install slurp-graph
|
|
34
|
+
|
|
35
|
+
# or with uv
|
|
36
|
+
uv add slurp-graph
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
> PyPI package: `slurp-graph` — CLI command: `slurp`
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Quickstart
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
slurp "auth flow" --graph graph.json --budget 4000
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
╭─ Slurp — Subgraph for: "auth flow" (budget: 4,000 tokens) ──────────────╮
|
|
51
|
+
│ Selected 5/2111 nodes · 847/4,000 tokens used (21.2%) │
|
|
52
|
+
╰───────────────────────────────────────────────────────────────────────────╯
|
|
53
|
+
|
|
54
|
+
## Relevant Nodes
|
|
55
|
+
|
|
56
|
+
### authenticate_user (function) · score: 0.94
|
|
57
|
+
Validates user credentials and returns JWT token.
|
|
58
|
+
→ File: src/auth/service.py
|
|
59
|
+
|
|
60
|
+
### JWTMiddleware (class) · score: 0.87
|
|
61
|
+
Intercepts HTTP requests and validates Authorization header.
|
|
62
|
+
→ File: src/middleware/jwt.py
|
|
63
|
+
|
|
64
|
+
### hash_password (function) · score: 0.71
|
|
65
|
+
Hashes password using bcrypt with a cost factor of 12.
|
|
66
|
+
→ File: src/auth/utils.py
|
|
67
|
+
|
|
68
|
+
## Key Relationships
|
|
69
|
+
- JWTMiddleware → calls → authenticate_user
|
|
70
|
+
- authenticate_user → calls → hash_password
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
💡 2106 additional connected nodes available — increase --budget to include them
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Add `--inject-code` to embed the actual function body next to each node:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
slurp "auth flow" --graph graph.json --budget 4000 --inject-code
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
````
|
|
83
|
+
### authenticate_user (function) · score: 0.94
|
|
84
|
+
Validates user credentials and returns JWT token.
|
|
85
|
+
→ File: src/auth/service.py
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
def authenticate_user(username: str, password: str) -> dict | None:
|
|
89
|
+
user = db.query(User).filter_by(username=username).first()
|
|
90
|
+
if not user or not bcrypt.checkpw(password.encode(), user.password_hash):
|
|
91
|
+
return None
|
|
92
|
+
return {"token": jwt.encode({"sub": user.id}, SECRET_KEY)}
|
|
93
|
+
```
|
|
94
|
+
````
|
|
95
|
+
|
|
96
|
+
Pipe the output directly into your LLM prompt, save it to a file, or use `slurp export` to format it as a ready-to-paste system prompt block.
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## Commands
|
|
101
|
+
|
|
102
|
+
### `slurp QUERY`
|
|
103
|
+
|
|
104
|
+
The main command. Scores all graph nodes against your query and selects the optimal subgraph within the token budget.
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
slurp "auth flow" --graph graph.json --budget 4000
|
|
108
|
+
slurp "payment processing" --format json
|
|
109
|
+
slurp "JWT validation" --explain
|
|
110
|
+
slurp "database schema" --inject-code --min-score 0.3
|
|
111
|
+
slurp "prisma models" --backend openai
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
| Flag | Default | Description |
|
|
115
|
+
|---|---|---|
|
|
116
|
+
| `--graph`, `-g` | auto-discover | Path to `graph.json`. |
|
|
117
|
+
| `--budget`, `-b` | `4000` | Token budget for subgraph selection. |
|
|
118
|
+
| `--format`, `-f` | `markdown` | Output format: `markdown`, `json`, or `yaml`. |
|
|
119
|
+
| `--model`, `-m` | `cl100k_base` | Tiktoken encoding for token counting. |
|
|
120
|
+
| `--explain` | off | Print per-node score breakdown: final / structural / semantic. |
|
|
121
|
+
| `--no-audit` | off | Skip writing to `.slurp/audit.jsonl`. |
|
|
122
|
+
| `--neighbor-decay` | `0.7` | Score multiplier applied to neighbors of each selected node. |
|
|
123
|
+
| `--min-score` | `0.15` | Minimum relevance score; nodes below this are excluded before selection. |
|
|
124
|
+
| `--viz` | off | Open an interactive graph visualization in the browser. |
|
|
125
|
+
| `--ignore-file` | `.slurpignore` | Path to node exclusion rules. |
|
|
126
|
+
| `--backend` | `tfidf` | Scoring backend: `tfidf` (default), `openai`, or `anthropic`. |
|
|
127
|
+
| `--inject-code` | off | Embed source code blocks for each selected node (requires ≤30 nodes). |
|
|
128
|
+
| `--project-root` | graph dir | Root directory for resolving `source_file` paths. |
|
|
129
|
+
|
|
130
|
+
**Auto-discovery** (when `--graph` is omitted):
|
|
131
|
+
|
|
132
|
+
1. `./graph.json`
|
|
133
|
+
2. `./graphify-out/graph.json`
|
|
134
|
+
3. `./.graphify/graph.json`
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
### `slurp stats`
|
|
139
|
+
|
|
140
|
+
Print node and edge counts for a graph file.
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
slurp stats --graph graph.json
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
Graph: graph.json
|
|
148
|
+
Nodes: 2111
|
|
149
|
+
Edges: 4823
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
### `slurp audit`
|
|
155
|
+
|
|
156
|
+
Show the history of queries logged to `.slurp/audit.jsonl`, plus the most frequently selected nodes.
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
slurp audit
|
|
160
|
+
slurp audit --top-nodes 20
|
|
161
|
+
slurp audit --audit-dir /custom/.slurp
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Every query is appended as a JSON line (unless `--no-audit` is passed). Useful for tracking which parts of your codebase an AI agent visits most.
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
### `slurp diff`
|
|
169
|
+
|
|
170
|
+
Compare two graph versions and report the impact of changes.
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
slurp diff old.json new.json
|
|
174
|
+
slurp diff old.json new.json --hops 2 --viz
|
|
175
|
+
slurp diff old.json new.json --budget 4000
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Reports added/removed/modified nodes and edges, computes an impact score based on centrality, and optionally opens a diff-colored visualization (green=added, red=removed, yellow=modified, grey=unchanged). Pass `--budget` to further select the most relevant affected nodes.
|
|
179
|
+
|
|
180
|
+
---
|
|
181
|
+
|
|
182
|
+
### `slurp export`
|
|
183
|
+
|
|
184
|
+
Export a context block ready to paste into an AI system prompt.
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
slurp export "auth flow" --format claude # <context> XML tags
|
|
188
|
+
slurp export "auth flow" --format chatgpt # [CODEBASE CONTEXT] block
|
|
189
|
+
slurp export "auth flow" --format claudemd # ## Codebase Context for CLAUDE.md
|
|
190
|
+
slurp export "auth flow" --output context.md
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
All three formats include query, nodes selected/total, tokens used/budget, and coverage %.
|
|
194
|
+
|
|
195
|
+
---
|
|
196
|
+
|
|
197
|
+
### `slurp serve`
|
|
198
|
+
|
|
199
|
+
Start an MCP stdio server (JSON-RPC 2.0) that exposes the `slurp_query` tool.
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
slurp serve --graph graph.json
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
See [MCP Integration](#mcp-integration) for configuration.
|
|
206
|
+
|
|
207
|
+
---
|
|
208
|
+
|
|
209
|
+
### `slurp benchmark`
|
|
210
|
+
|
|
211
|
+
Measure real token savings across queries and budgets.
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
slurp benchmark \
|
|
215
|
+
--graph graph.json \
|
|
216
|
+
--queries "auth flow" --queries "schema validation" \
|
|
217
|
+
--budget 2000 --budget 4000 --budget 8000
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Outputs a per-run table and aggregate stats: mean savings, p50/p90/p95, best/worst case, and precision (fraction of relevant nodes captured).
|
|
221
|
+
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
## Works with graphify
|
|
225
|
+
|
|
226
|
+
Slurp is the query layer for [`graphify`](https://github.com/CarlosVallejoRuiz/graphify). Run graphify on your codebase, point slurp at the output.
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
graphify . # generates graphify-out/graph.json
|
|
230
|
+
slurp "auth flow" --budget 4000 # auto-discovers graphify-out/graph.json
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
**Supported node fields:**
|
|
234
|
+
|
|
235
|
+
```json
|
|
236
|
+
{
|
|
237
|
+
"id": "authenticate_user",
|
|
238
|
+
"label": "authenticate_user",
|
|
239
|
+
"type": "function",
|
|
240
|
+
"description": "Validates credentials and returns JWT.",
|
|
241
|
+
"importance": 9,
|
|
242
|
+
"source_file": "src/auth/service.py",
|
|
243
|
+
"source_location": "L42"
|
|
244
|
+
}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
The `type`, `description`, `importance`, `source_file`, and `source_location` fields are optional but improve scoring and enable `--inject-code`. Any graph with `id` + `label` on nodes and `source`/`target` on edges will work.
|
|
248
|
+
|
|
249
|
+
Both `links` (graphify/NetworkX serialization) and `edges` are supported. Additional formats are auto-detected by extension:
|
|
250
|
+
|
|
251
|
+
| Extension | Format |
|
|
252
|
+
|---|---|
|
|
253
|
+
| `.json` | graphify or generic JSON |
|
|
254
|
+
| `.graphml` | GraphML (NetworkX / yEd / Gephi) |
|
|
255
|
+
| `.csv` | Neo4j export (nodes CSV + sibling relationships CSV) |
|
|
256
|
+
|
|
257
|
+
Use `slurp convert` or the `convert_graph()` API to export between formats.
|
|
258
|
+
|
|
259
|
+
---
|
|
260
|
+
|
|
261
|
+
## MCP Integration
|
|
262
|
+
|
|
263
|
+
Run slurp as an MCP server so Claude Code (or any MCP-compatible agent) can query the graph directly.
|
|
264
|
+
|
|
265
|
+
**`.mcp.json`:**
|
|
266
|
+
|
|
267
|
+
```json
|
|
268
|
+
{
|
|
269
|
+
"mcpServers": {
|
|
270
|
+
"slurp": {
|
|
271
|
+
"command": "/path/to/.venv/bin/slurp",
|
|
272
|
+
"args": ["serve", "--graph", "/path/to/graphify-out/graph.json"]
|
|
273
|
+
}
|
|
274
|
+
}
|
|
275
|
+
}
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
**Tool exposed:** `slurp_query(query: str, budget: int = 4000) → str`
|
|
279
|
+
|
|
280
|
+
Claude Code calls this automatically when it needs codebase context. The server runs over stdio and returns the formatted markdown subgraph — no HTTP, no ports.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
## .slurpignore
|
|
285
|
+
|
|
286
|
+
Exclude nodes by type, file path, or ID pattern. Create `.slurpignore` in your project root:
|
|
287
|
+
|
|
288
|
+
```
|
|
289
|
+
# Exclude documentation nodes
|
|
290
|
+
type:document
|
|
291
|
+
type:markdown
|
|
292
|
+
|
|
293
|
+
# Exclude test files
|
|
294
|
+
file:tests/**
|
|
295
|
+
file:**/*.test.ts
|
|
296
|
+
|
|
297
|
+
# Exclude generated code
|
|
298
|
+
id:generated_*
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
Pass a custom path with `--ignore-file path/to/.slurpignore`.
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
305
|
+
## Design decisions
|
|
306
|
+
|
|
307
|
+
**Power-iteration PageRank without numpy.** `nx.pagerank()` requires numpy. Slurp implements a 20-line pure-Python power-iteration algorithm (convergence: `Σ|rank_new − rank_old| < N × tol`). Same result, no heavy dependency.
|
|
308
|
+
|
|
309
|
+
**TF-IDF without scikit-learn.** Hand-rolled TF-IDF with smoothed IDF (`log((N+1)/(df+1)) + 1`) and cosine similarity. The tokenizer splits camelCase and snake_case, so `authenticate_user` scores on both `authenticate` and `user`. The `score_nodes()` interface is backend-agnostic — swap to real embeddings with `--backend openai` or `--backend anthropic` without touching any caller.
|
|
310
|
+
|
|
311
|
+
**YAML serializer without PyYAML.** `_yaml_scalar()` renders Python primitives as valid YAML scalars using `json.dumps()` for strings that need quoting (JSON string literals are valid YAML 1.1). No PyYAML dependency.
|
|
312
|
+
|
|
313
|
+
**`lru_cache` on the tiktoken encoder.** `tiktoken.get_encoding()` reads tokenizer data from disk on first call. Caching with `lru_cache(maxsize=8)` means repeated token-counting calls within a single run hit memory, not disk.
|
|
314
|
+
|
|
315
|
+
**`+0.3` score boost for `file_type == "code"` nodes (clamped to 1.0).** Documentation nodes compete unfairly with code in technical queries. The boost is bounded so it cannot override a genuinely high structural+semantic score.
|
|
316
|
+
|
|
317
|
+
**`--inject-code` capped at 30 nodes.** Code blocks are 50–200 tokens each. At 30 nodes, that's up to 6,000 extra tokens — manageable. At 200 nodes it would explode the context budget. The cap is enforced in both the CLI (warning message) and `inject_code()` (hard guard), so the formatter never receives oversized input.
|
|
318
|
+
|
|
319
|
+
---
|
|
320
|
+
|
|
321
|
+
## Roadmap
|
|
322
|
+
|
|
323
|
+
- ✅ **v0.1.0** — `loader`, `scorer`, `budget`, `formatter`, `audit` — core pipeline, full tests, `slurp QUERY` + `slurp stats`
|
|
324
|
+
- ✅ **v0.2.0** — `--explain`, `.slurpignore`, `--viz` interactive HTML, `--min-score`, camelCase/snake_case tokenizer, `--neighbor-decay`
|
|
325
|
+
- ✅ **v0.3.0** — `slurp serve` (MCP stdio), `slurp diff`, `slurp export` (claude/chatgpt/claudemd), PyPI publish as `slurp-graph`
|
|
326
|
+
- ✅ **v0.4.0** — `--backend openai|anthropic` (optional embeddings), `slurp benchmark`, GraphML + Neo4j CSV loader, `convert_graph()`
|
|
327
|
+
- ✅ **v0.5.0** — `--inject-code`: extract real function bodies from source files and embed them in the context output
|
|
328
|
+
|
|
329
|
+
---
|
|
330
|
+
|
|
331
|
+
## License
|
|
332
|
+
|
|
333
|
+
MIT © [Juan Carlos Vallejo Ruiz](https://github.com/CarlosVallejoRuiz)
|