jorgex-stack 1.0.8 → 1.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # JorgeX Stack
2
2
 
3
- Portable multi-agent harness: one configuration source — 15 agents, 18 skills, hooks, persistent memory ([Engram](https://github.com/Gentleman-Programming/engram)), MCPs, and system prompt — installable with one command in **Claude Code**, **Codex CLI**, and **OpenCode**.
3
+ Portable multi-agent harness: one configuration source — 15 agents, 17 skills, hooks, persistent memory ([Engram](https://github.com/Gentleman-Programming/engram)), MCPs, and system prompt — installable with one command in **Claude Code**, **Codex CLI**, and **OpenCode**.
4
4
 
5
5
  > Inspired by [gentle-ai](https://github.com/Gentleman-Programming/gentle-ai), rebuilt for the JorgeX stack.
6
6
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "jorgex-stack",
3
- "version": "1.0.8",
3
+ "version": "1.0.9",
4
4
  "description": "Harness multi-agente portable: instala la config JorgeX (agentes, skills, hooks, Engram, MCPs) en Claude Code, Codex CLI y OpenCode",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/upstreams.json CHANGED
@@ -32,12 +32,6 @@
32
32
  "path": "skills/find-skills",
33
33
  "commit": "be0dd25b4a8665894a56f45ef582cc02ca802c39"
34
34
  },
35
- "graphify": {
36
- "source": "github:safishamsi/graphify",
37
- "kind": "release",
38
- "version": "0.7.16",
39
- "commit": "cce26730212baab6d92ce5f390ef5aae56268f31"
40
- },
41
35
  "mcp-builder": {
42
36
  "source": "github:anthropics/skills",
43
37
  "path": "skills/mcp-builder",
@@ -1 +0,0 @@
1
- 0.7.16
@@ -1,1319 +0,0 @@
1
- ---
2
- name: graphify
3
- description: "any input (code, docs, papers, images) → knowledge graph → clustered communities → HTML + JSON + audit report. Use when user asks any question about a codebase, project content, architecture, or file relationships — especially if graphify-out/ exists. Provides persistent graph with god nodes, community detection, and BFS/DFS query tools."
4
- trigger: /graphify
5
- ---
6
-
7
- # /graphify
8
-
9
- Turn any folder of files into a navigable knowledge graph with community detection, an honest audit trail, and three outputs: interactive HTML, GraphRAG-ready JSON, and a plain-language GRAPH_REPORT.md.
10
-
11
- ## Usage
12
-
13
- ```
14
- /graphify # full pipeline on current directory → Obsidian vault
15
- /graphify <path> # full pipeline on specific path
16
- /graphify <path> --mode deep # thorough extraction, richer INFERRED edges
17
- /graphify <path> --update # incremental - re-extract only new/changed files
18
- /graphify <path> --cluster-only # rerun clustering on existing graph
19
- /graphify <path> --no-viz # skip visualization, just report + JSON
20
- /graphify <path> --html # (HTML is generated by default - this flag is a no-op)
21
- /graphify <path> --svg # also export graph.svg (embeds in Notion, GitHub)
22
- /graphify <path> --graphml # export graph.graphml (Gephi, yEd)
23
- /graphify <path> --neo4j # generate graphify-out/cypher.txt for Neo4j
24
- /graphify <path> --neo4j-push bolt://localhost:7687 # push directly to Neo4j
25
- /graphify <path> --mcp # start MCP stdio server for agent access
26
- /graphify <path> --watch # watch folder, auto-rebuild on code changes (no LLM needed)
27
- /graphify add <url> # fetch URL, save to ./raw, update graph
28
- /graphify add <url> --author "Name" # tag who wrote it
29
- /graphify add <url> --contributor "Name" # tag who added it to the corpus
30
- /graphify query "<question>" # BFS traversal - broad context
31
- /graphify query "<question>" --dfs # DFS - trace a specific path
32
- /graphify query "<question>" --budget 1500 # cap answer at N tokens
33
- /graphify path "AuthModule" "Database" # shortest path between two concepts
34
- /graphify explain "SwinTransformer" # plain-language explanation of a node
35
- ```
36
-
37
- ## What graphify is for
38
-
39
- graphify is built around Andrej Karpathy's /raw folder workflow: drop anything into a folder - papers, tweets, screenshots, code, notes - and get a structured knowledge graph that shows you what you didn't know was connected.
40
-
41
- Three things it does that your AI assistant alone cannot:
42
- 1. **Persistent graph** - relationships are stored in `graphify-out/graph.json` and survive across sessions. Ask questions weeks later without re-reading everything.
43
- 2. **Honest audit trail** - every edge is tagged EXTRACTED, INFERRED, or AMBIGUOUS. You know what was found vs invented.
44
- 3. **Cross-document surprise** - community detection finds connections between concepts in different files that you would never think to ask about directly.
45
-
46
- Use it for:
47
- - A codebase you're new to (understand architecture before touching anything)
48
- - A reading list (papers + tweets + notes → one navigable graph)
49
- - A research corpus (citation graph + concept graph in one)
50
- - Your personal /raw folder (drop everything in, let it grow, query it)
51
-
52
- ## What You Must Do When Invoked
53
-
54
- If the user invoked `/graphify --help` or `/graphify -h` (with no other arguments), print the contents of the `## Usage` section above verbatim and stop. Do not run any commands, do not detect files, do not default the path to `.`. Just print the Usage block and return.
55
-
56
- If no path was given, use `.` (current directory). Do not ask the user for a path.
57
-
58
- Follow these steps in order. Do not skip steps.
59
-
60
- ### Step 1 - Ensure graphify is installed
61
-
62
- ```bash
63
- # Detect the correct Python interpreter (handles pipx, venv, system installs)
64
- GRAPHIFY_BIN=$(which graphify 2>/dev/null)
65
- if [ -n "$GRAPHIFY_BIN" ]; then
66
- PYTHON=$(head -1 "$GRAPHIFY_BIN" | tr -d '#!')
67
- case "$PYTHON" in
68
- *[!a-zA-Z0-9/_.-]*) PYTHON="python3" ;;
69
- esac
70
- else
71
- PYTHON="python3"
72
- fi
73
- "$PYTHON" -c "import graphify" 2>/dev/null || "$PYTHON" -m pip install graphifyy -q 2>/dev/null || "$PYTHON" -m pip install graphifyy -q --break-system-packages 2>&1 | tail -3
74
- # Write interpreter path for all subsequent steps
75
- mkdir -p graphify-out
76
- "$PYTHON" -c "import sys; open('graphify-out/.graphify_python', 'w').write(sys.executable)"
77
- # Force UTF-8 I/O on Windows (prevents garbled CJK/non-ASCII output)
78
- export PYTHONUTF8=1
79
- ```
80
-
81
- If the import succeeds, print nothing and move straight to Step 2.
82
-
83
- **In every subsequent bash block, replace `python3` with `$(cat graphify-out/.graphify_python)` to use the correct interpreter.**
84
-
85
- ### Step 2 - Detect files
86
-
87
- ```bash
88
- $(cat graphify-out/.graphify_python) -c "
89
- import json
90
- from graphify.detect import detect
91
- from pathlib import Path
92
- result = detect(Path('INPUT_PATH'))
93
- print(json.dumps(result))
94
- " > graphify-out/.graphify_detect.json
95
- ```
96
-
97
- Replace INPUT_PATH with the actual path the user provided. Do NOT cat or print the JSON - read it silently and present a clean summary instead:
98
-
99
- ```
100
- Corpus: X files · ~Y words
101
- code: N files (.py .ts .go ...)
102
- docs: N files (.md .txt ...)
103
- papers: N files (.pdf ...)
104
- images: N files
105
- video: N files (.mp4 .mp3 ...)
106
- ```
107
-
108
- Omit any category with 0 files from the summary.
109
-
110
- Then act on it:
111
- - If `total_files` is 0: stop with "No supported files found in [path]."
112
- - If `skipped_sensitive` is non-empty: mention file count skipped, not the file names.
113
- - If `total_words` > 2,000,000 OR `total_files` > 200: show the warning and the top 5 subdirectories by file count, then ask which subfolder to run on. Wait for the user's answer before proceeding.
114
- - Otherwise: proceed directly to Step 2.5 if video files were detected, or Step 3 if not.
115
-
116
- ### Step 2.5 - Transcribe video / audio files (only if video files detected)
117
-
118
- Skip this step entirely if `detect` returned zero `video` files.
119
-
120
- Video and audio files cannot be read directly. Transcribe them to text first, then treat the transcripts as doc files in Step 3.
121
-
122
- **Strategy:** Read the god nodes from the detect output or analysis file. You are already a language model - write a one-sentence domain hint yourself from those labels. Then pass it to Whisper as the initial prompt. No separate API call needed.
123
-
124
- **However**, if the corpus has *only* video files and no other docs/code, use the generic fallback prompt: `"Use proper punctuation and paragraph breaks."`
125
-
126
- **Step 1 - Write the Whisper prompt yourself.**
127
-
128
- Read the top god node labels from detect output or analysis, then compose a short domain hint sentence, for example:
129
-
130
- - Labels: `transformer, attention, encoder, decoder` -> `"Machine learning research on transformer architectures and attention mechanisms. Use proper punctuation and paragraph breaks."`
131
- - Labels: `kubernetes, deployment, pod, helm` -> `"DevOps discussion about Kubernetes deployments and Helm charts. Use proper punctuation and paragraph breaks."`
132
-
133
- Set it as `GRAPHIFY_WHISPER_PROMPT` in the environment before running the transcription command.
134
-
135
- **Step 2 - Transcribe:**
136
-
137
- ```bash
138
- $(cat graphify-out/.graphify_python) -c "
139
- import json, os
140
- from pathlib import Path
141
- from graphify.transcribe import transcribe_all
142
-
143
- detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
144
- video_files = detect.get('files', {}).get('video', [])
145
- prompt = os.environ.get('GRAPHIFY_WHISPER_PROMPT', 'Use proper punctuation and paragraph breaks.')
146
-
147
- transcript_paths = transcribe_all(video_files, initial_prompt=prompt)
148
- print(json.dumps(transcript_paths))
149
- " > graphify-out/.graphify_transcripts.json
150
- ```
151
-
152
- After transcription:
153
- - Read the transcript paths from `graphify-out/.graphify_transcripts.json`
154
- - Add them to the docs list before dispatching semantic subagents in Step 3B
155
- - Print how many transcripts were created: `Transcribed N video file(s) -> treating as docs`
156
- - If transcription fails for a file, print a warning and continue with the rest
157
-
158
- **Whisper model:** Default is `base`. If the user passed `--whisper-model <name>`, set `GRAPHIFY_WHISPER_MODEL=<name>` in the environment before running the command above.
159
-
160
- ### Step 3 - Extract entities and relationships
161
-
162
- **Before starting:** note whether `--mode deep` was given. You must pass `DEEP_MODE=true` to every subagent in Step B2 if it was. Track this from the original invocation - do not lose it.
163
-
164
- This step has two parts: **structural extraction** (deterministic, free) and **semantic extraction** (your AI model, costs tokens).
165
-
166
- **Run Part A (AST) and Part B (semantic) in parallel. Dispatch all semantic subagents AND start AST extraction in the same message. Both can run simultaneously since they operate on different file types. Merge results in Part C as before.**
167
-
168
- Note: Parallelizing AST + semantic saves 5-15s on large corpora. AST is deterministic and fast; start it while subagents are processing docs/papers.
169
-
170
- #### Part A - Structural extraction for code files
171
-
172
- For any code files detected, run AST extraction in parallel with Part B subagents:
173
-
174
- ```bash
175
- $(cat graphify-out/.graphify_python) -c "
176
- import sys, json
177
- from graphify.extract import collect_files, extract
178
- from pathlib import Path
179
- import json
180
-
181
- code_files = []
182
- detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
183
- for f in detect.get('files', {}).get('code', []):
184
- code_files.extend(collect_files(Path(f)) if Path(f).is_dir() else [Path(f)])
185
-
186
- if code_files:
187
- result = extract(code_files)
188
- Path('graphify-out/.graphify_ast.json').write_text(json.dumps(result, indent=2))
189
- print(f'AST: {len(result[\"nodes\"])} nodes, {len(result[\"edges\"])} edges')
190
- else:
191
- Path('graphify-out/.graphify_ast.json').write_text(json.dumps({'nodes':[],'edges':[],'input_tokens':0,'output_tokens':0}))
192
- print('No code files - skipping AST extraction')
193
- "
194
- ```
195
-
196
- #### Part B - Semantic extraction (parallel subagents)
197
-
198
- **Fast path:** If detection found zero docs, papers, and images (code-only corpus), skip Part B entirely and go straight to Part C. AST handles code - there is nothing for semantic subagents to do.
199
-
200
- **MANDATORY: You MUST use the Agent tool here. Reading files yourself one-by-one is forbidden - it is 5-10x slower. If you do not use the Agent tool you are doing this wrong.**
201
-
202
- Before dispatching subagents, print a timing estimate:
203
- - Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
204
- - Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
205
- - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
206
- - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
207
-
208
- **Step B0 - Check extraction cache first**
209
-
210
- Before dispatching any subagents, check which files already have cached extraction results:
211
-
212
- ```bash
213
- $(cat graphify-out/.graphify_python) -c "
214
- import json
215
- from graphify.cache import check_semantic_cache
216
- from pathlib import Path
217
-
218
- detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
219
- all_files = [f for files in detect['files'].values() for f in files]
220
-
221
- cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files)
222
-
223
- if cached_nodes or cached_edges or cached_hyperedges:
224
- Path('graphify-out/.graphify_cached.json').write_text(json.dumps({'nodes': cached_nodes, 'edges': cached_edges, 'hyperedges': cached_hyperedges}))
225
- Path('graphify-out/.graphify_uncached.txt').write_text('\n'.join(uncached))
226
- print(f'Cache: {len(all_files)-len(uncached)} files hit, {len(uncached)} files need extraction')
227
- "
228
- ```
229
-
230
- Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt`. If all files are cached, skip to Part C directly.
231
-
232
- **Step B1 - Split into chunks**
233
-
234
- Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
235
-
236
- **Step B2 - Dispatch ALL subagents in a single message (OpenCode)**
237
-
238
- > **OpenCode platform:** Uses `@mention` dispatch instead of the Agent tool. All mentions in a single message run in parallel.
239
-
240
- Dispatch one `@mention` per chunk — ALL in the same response:
241
-
242
- ```
243
- @agent Chunk CHUNK_NUM of TOTAL_CHUNKS: [extraction prompt below with FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE substituted]
244
-
245
- @agent Chunk 2 of TOTAL_CHUNKS: [next chunk]
246
- ```
247
-
248
- Wait for all agents to return. Parse each response as JSON. Accumulate nodes/edges/hyperedges across all results and write to `graphify-out/.graphify_semantic_new.json`.
249
-
250
- The extraction prompt each agent receives (substitute FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE):
251
-
252
- ```
253
- You are a graphify extraction subagent. Read the files listed and extract a knowledge graph fragment.
254
- Output ONLY valid JSON matching the schema below - no explanation, no markdown fences, no preamble.
255
-
256
- Files (chunk CHUNK_NUM of TOTAL_CHUNKS):
257
- FILE_LIST
258
-
259
- Rules:
260
- - EXTRACTED: relationship explicit in source (import, call, citation, "see §3.2")
261
- - INFERRED: reasonable inference (shared data structure, implied dependency)
262
- - AMBIGUOUS: uncertain - flag for review, do not omit
263
-
264
- Code files: focus on semantic edges AST cannot find (call relationships, shared data, arch patterns).
265
- Do not re-extract imports - AST already has those.
266
- Doc/paper files: extract named concepts, entities, citations. For rationale (WHY decisions were made, trade-offs, design intent): store as a `rationale` attribute on the relevant concept node — do NOT create a separate rationale node or fragment node. Only create a node for something that is itself a named entity or concept. Use `file_type:"rationale"` for concept-like nodes (ideas, principles, mechanisms, design patterns). Do NOT invent file_types like `concept` — valid values are only `code|document|paper|image|rationale`.
267
- Code files: when adding `calls` edges, source MUST be the caller (the function/class doing the calling), target MUST be the callee. Never reverse this direction.
268
- Image files: use vision to understand what the image IS - do not just OCR.
269
- UI screenshot: layout patterns, design decisions, key elements, purpose.
270
- Chart: metric, trend/insight, data source.
271
- Tweet/post: claim as node, author, concepts mentioned.
272
- Diagram: components and connections.
273
- Research figure: what it demonstrates, method, result.
274
- Handwritten/whiteboard: ideas and arrows, mark uncertain readings AMBIGUOUS.
275
-
276
- DEEP_MODE (if --mode deep was given): be aggressive with INFERRED edges - indirect deps,
277
- shared assumptions, latent couplings. Mark uncertain ones AMBIGUOUS instead of omitting.
278
-
279
- Semantic similarity: if two concepts in this chunk solve the same problem or represent the same idea without any structural link (no import, no call, no citation), add a `semantically_similar_to` edge marked INFERRED with a confidence_score reflecting how similar they are (0.6-0.95). Examples:
280
- - Two functions that both validate user input but never call each other
281
- - A class in code and a concept in a paper that describe the same algorithm
282
- - Two error types that handle the same failure mode differently
283
- Only add these when the similarity is genuinely non-obvious and cross-cutting. Do not add them for trivially similar things.
284
-
285
- Hyperedges: if 3 or more nodes clearly participate together in a shared concept, flow, or pattern that is not captured by pairwise edges alone, add a hyperedge to a top-level `hyperedges` array. Examples:
286
- - All classes that implement a common protocol or interface
287
- - All functions in an authentication flow (even if they don't all call each other)
288
- - All concepts from a paper section that form one coherent idea
289
- Use sparingly — only when the group relationship adds information beyond the pairwise edges. Maximum 3 hyperedges per chunk.
290
-
291
- If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, author,
292
- contributor onto every node from that file.
293
-
294
- confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
295
- - EXTRACTED edges: confidence_score = 1.0 always
296
- - INFERRED edges: reason about each edge individually.
297
- Direct structural evidence (shared data structure, clear dependency): 0.8-0.9.
298
- Reasonable inference with some uncertainty: 0.6-0.7.
299
- Weak or speculative: 0.4-0.5. Most edges should be 0.6-0.9, not 0.5.
300
- - AMBIGUOUS edges: 0.1-0.3
301
-
302
- Output exactly this JSON (no other text):
303
- {"nodes":[{"id":"filestem_entityname","label":"Human Readable Name","file_type":"code|document|paper|image|rationale","source_file":"relative/path","source_location":null,"source_url":null,"captured_at":null,"author":null,"contributor":null}],"edges":[{"source":"node_id","target":"node_id","relation":"calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to|rationale_for","confidence":"EXTRACTED|INFERRED|AMBIGUOUS","confidence_score":1.0,"source_file":"relative/path","source_location":null,"weight":1.0}],"hyperedges":[{"id":"snake_case_id","label":"Human Readable Label","nodes":["node_id1","node_id2","node_id3"],"relation":"participate_in|implement|form","confidence":"EXTRACTED|INFERRED","confidence_score":0.75,"source_file":"relative/path"}],"input_tokens":0,"output_tokens":0}
304
- ```
305
-
306
- **Step B3 - Collect, cache, and merge**
307
-
308
- Wait for all subagents. For each result:
309
- - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
310
- - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
311
- - If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
312
- - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
313
-
314
- If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
315
-
316
- Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
317
- ```bash
318
- $(cat graphify-out/.graphify_python) -c "
319
- import json, glob
320
- from pathlib import Path
321
-
322
- chunks = sorted(glob.glob('graphify-out/.graphify_chunk_*.json'))
323
- all_nodes, all_edges, all_hyperedges = [], [], []
324
- total_in, total_out = 0, 0
325
- for c in chunks:
326
- d = json.loads(Path(c).read_text())
327
- all_nodes += d.get('nodes', [])
328
- all_edges += d.get('edges', [])
329
- all_hyperedges += d.get('hyperedges', [])
330
- total_in += d.get('input_tokens', 0)
331
- total_out += d.get('output_tokens', 0)
332
- Path('graphify-out/.graphify_semantic_new.json').write_text(json.dumps({
333
- 'nodes': all_nodes, 'edges': all_edges, 'hyperedges': all_hyperedges,
334
- 'input_tokens': total_in, 'output_tokens': total_out,
335
- }, indent=2))
336
- print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens')
337
- "
338
- ```
339
-
340
- Save new results to cache:
341
- ```bash
342
- $(cat graphify-out/.graphify_python) -c "
343
- import json
344
- from graphify.cache import save_semantic_cache
345
- from pathlib import Path
346
-
347
- new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text()) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
348
- saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []))
349
- print(f'Cached {saved} files')
350
- "
351
- ```
352
-
353
- Merge cached + new results into `graphify-out/.graphify_semantic.json`:
354
- ```bash
355
- $(cat graphify-out/.graphify_python) -c "
356
- import json
357
- from pathlib import Path
358
-
359
- cached = json.loads(Path('graphify-out/.graphify_cached.json').read_text()) if Path('graphify-out/.graphify_cached.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
360
- new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text()) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
361
-
362
- all_nodes = cached['nodes'] + new.get('nodes', [])
363
- all_edges = cached['edges'] + new.get('edges', [])
364
- all_hyperedges = cached.get('hyperedges', []) + new.get('hyperedges', [])
365
- seen = set()
366
- deduped = []
367
- for n in all_nodes:
368
- if n['id'] not in seen:
369
- seen.add(n['id'])
370
- deduped.append(n)
371
-
372
- merged = {
373
- 'nodes': deduped,
374
- 'edges': all_edges,
375
- 'hyperedges': all_hyperedges,
376
- 'input_tokens': new.get('input_tokens', 0),
377
- 'output_tokens': new.get('output_tokens', 0),
378
- }
379
- Path('graphify-out/.graphify_semantic.json').write_text(json.dumps(merged, indent=2))
380
- print(f'Extraction complete - {len(deduped)} nodes, {len(all_edges)} edges ({len(cached[\"nodes\"])} from cache, {len(new.get(\"nodes\",[]))} new)')
381
- "
382
- ```
383
- Clean up temp files: `rm -f graphify-out/.graphify_cached.json graphify-out/.graphify_uncached.txt graphify-out/.graphify_semantic_new.json`
384
-
385
- #### Part C - Merge AST + semantic into final extraction
386
-
387
- ```bash
388
- $(cat graphify-out/.graphify_python) -c "
389
- import sys, json
390
- from pathlib import Path
391
-
392
- ast = json.loads(Path('graphify-out/.graphify_ast.json').read_text())
393
- sem = json.loads(Path('graphify-out/.graphify_semantic.json').read_text())
394
-
395
- # Merge: AST nodes first, semantic nodes deduplicated by id
396
- seen = {n['id'] for n in ast['nodes']}
397
- merged_nodes = list(ast['nodes'])
398
- for n in sem['nodes']:
399
- if n['id'] not in seen:
400
- merged_nodes.append(n)
401
- seen.add(n['id'])
402
-
403
- merged_edges = ast['edges'] + sem['edges']
404
- merged_hyperedges = sem.get('hyperedges', [])
405
- merged = {
406
- 'nodes': merged_nodes,
407
- 'edges': merged_edges,
408
- 'hyperedges': merged_hyperedges,
409
- 'input_tokens': sem.get('input_tokens', 0),
410
- 'output_tokens': sem.get('output_tokens', 0),
411
- }
412
- Path('graphify-out/.graphify_extract.json').write_text(json.dumps(merged, indent=2))
413
- total = len(merged_nodes)
414
- edges = len(merged_edges)
415
- print(f'Merged: {total} nodes, {edges} edges ({len(ast[\"nodes\"])} AST + {len(sem[\"nodes\"])} semantic)')
416
- "
417
- ```
418
-
419
- ### Step 4 - Build graph, cluster, analyze, generate outputs
420
-
421
- ```bash
422
- mkdir -p graphify-out
423
- $(cat graphify-out/.graphify_python) -c "
424
- import sys, json
425
- from graphify.build import build_from_json
426
- from graphify.cluster import cluster, score_all
427
- from graphify.analyze import god_nodes, surprising_connections, suggest_questions
428
- from graphify.report import generate
429
- from graphify.export import to_json
430
- from pathlib import Path
431
-
432
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
433
- detection = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
434
-
435
- G = build_from_json(extraction)
436
- communities = cluster(G)
437
- cohesion = score_all(G, communities)
438
- tokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}
439
- gods = god_nodes(G)
440
- surprises = surprising_connections(G, communities)
441
- labels = {cid: 'Community ' + str(cid) for cid in communities}
442
- # Placeholder questions - regenerated with real labels in Step 5
443
- questions = suggest_questions(G, communities, labels)
444
-
445
- report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, 'INPUT_PATH', suggested_questions=questions)
446
- Path('graphify-out/GRAPH_REPORT.md').write_text(report)
447
- to_json(G, communities, 'graphify-out/graph.json')
448
-
449
- analysis = {
450
- 'communities': {str(k): v for k, v in communities.items()},
451
- 'cohesion': {str(k): v for k, v in cohesion.items()},
452
- 'gods': gods,
453
- 'surprises': surprises,
454
- 'questions': questions,
455
- }
456
- Path('graphify-out/.graphify_analysis.json').write_text(json.dumps(analysis, indent=2))
457
- if G.number_of_nodes() == 0:
458
- print('ERROR: Graph is empty - extraction produced no nodes.')
459
- print('Possible causes: all files were skipped, binary-only corpus, or extraction failed.')
460
- raise SystemExit(1)
461
- print(f'Graph: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges, {len(communities)} communities')
462
- "
463
- ```
464
-
465
- If this step prints `ERROR: Graph is empty`, stop and tell the user what happened - do not proceed to labeling or visualization.
466
-
467
- Replace INPUT_PATH with the actual path.
468
-
469
- ### Step 5 - Label communities
470
-
471
- Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading").
472
-
473
- Then regenerate the report and save the labels for the visualizer:
474
-
475
- ```bash
476
- $(cat graphify-out/.graphify_python) -c "
477
- import sys, json
478
- from graphify.build import build_from_json
479
- from graphify.cluster import score_all
480
- from graphify.analyze import god_nodes, surprising_connections, suggest_questions
481
- from graphify.report import generate
482
- from pathlib import Path
483
-
484
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
485
- detection = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
486
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
487
-
488
- G = build_from_json(extraction)
489
- communities = {int(k): v for k, v in analysis['communities'].items()}
490
- cohesion = {int(k): v for k, v in analysis['cohesion'].items()}
491
- tokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}
492
-
493
- # LABELS - replace these with the names you chose above
494
- labels = LABELS_DICT
495
-
496
- # Regenerate questions with real community labels (labels affect question phrasing)
497
- questions = suggest_questions(G, communities, labels)
498
-
499
- report = generate(G, communities, cohesion, labels, analysis['gods'], analysis['surprises'], detection, tokens, 'INPUT_PATH', suggested_questions=questions)
500
- Path('graphify-out/GRAPH_REPORT.md').write_text(report)
501
- Path('graphify-out/.graphify_labels.json').write_text(json.dumps({str(k): v for k, v in labels.items()}))
502
- print('Report updated with community labels')
503
- "
504
- ```
505
-
506
- Replace `LABELS_DICT` with the actual dict you constructed (e.g. `{0: "Attention Mechanism", 1: "Training Pipeline"}`).
507
- Replace INPUT_PATH with the actual path.
508
-
509
- ### Step 6 - Generate Obsidian vault (opt-in) + HTML
510
-
511
- **Generate HTML always** (unless `--no-viz`). **Obsidian vault only if `--obsidian` was explicitly given** — skip it otherwise, it generates one file per node.
512
-
513
- If `--obsidian` was given:
514
-
515
- ```bash
516
- $(cat graphify-out/.graphify_python) -c "
517
- import sys, json
518
- from graphify.build import build_from_json
519
- from graphify.export import to_obsidian, to_canvas
520
- from pathlib import Path
521
-
522
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
523
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
524
- labels_raw = json.loads(Path('graphify-out/.graphify_labels.json').read_text()) if Path('graphify-out/.graphify_labels.json').exists() else {}
525
-
526
- G = build_from_json(extraction)
527
- communities = {int(k): v for k, v in analysis['communities'].items()}
528
- cohesion = {int(k): v for k, v in analysis['cohesion'].items()}
529
- labels = {int(k): v for k, v in labels_raw.items()}
530
-
531
- n = to_obsidian(G, communities, 'graphify-out/obsidian', community_labels=labels or None, cohesion=cohesion)
532
- print(f'Obsidian vault: {n} notes in graphify-out/obsidian/')
533
-
534
- to_canvas(G, communities, 'graphify-out/obsidian/graph.canvas', community_labels=labels or None)
535
- print('Canvas: graphify-out/obsidian/graph.canvas - open in Obsidian for structured community layout')
536
- print()
537
- print('Open graphify-out/obsidian/ as a vault in Obsidian.')
538
- print(' Graph view - nodes colored by community (set automatically)')
539
- print(' graph.canvas - structured layout with communities as groups')
540
- print(' _COMMUNITY_* - overview notes with cohesion scores and dataview queries')
541
- "
542
- ```
543
-
544
- Generate the HTML graph (always, unless `--no-viz`):
545
-
546
- ```bash
547
- $(cat graphify-out/.graphify_python) -c "
548
- import sys, json
549
- from graphify.build import build_from_json
550
- from graphify.export import to_html
551
- from pathlib import Path
552
-
553
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
554
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
555
- labels_raw = json.loads(Path('graphify-out/.graphify_labels.json').read_text()) if Path('graphify-out/.graphify_labels.json').exists() else {}
556
-
557
- G = build_from_json(extraction)
558
- communities = {int(k): v for k, v in analysis['communities'].items()}
559
- labels = {int(k): v for k, v in labels_raw.items()}
560
-
561
- NODE_LIMIT = 5000
562
- if G.number_of_nodes() > NODE_LIMIT:
563
- from collections import Counter
564
- print(f'Graph has {G.number_of_nodes()} nodes (above {NODE_LIMIT} limit). Building aggregated community view...')
565
- node_to_community = {nid: cid for cid, members in communities.items() for nid in members}
566
- import networkx as nx_meta
567
- meta = nx_meta.Graph()
568
- for cid, members in communities.items():
569
- meta.add_node(str(cid), label=labels.get(cid, f'Community {cid}'))
570
- edge_counts = Counter()
571
- for u, v in G.edges():
572
- cu, cv = node_to_community.get(u), node_to_community.get(v)
573
- if cu is not None and cv is not None and cu != cv:
574
- edge_counts[(min(cu, cv), max(cu, cv))] += 1
575
- for (cu, cv), w in edge_counts.items():
576
- meta.add_edge(str(cu), str(cv), weight=w, relation=f'{w} cross-community edges', confidence='AGGREGATED')
577
- if meta.number_of_nodes() > 1:
578
- meta_communities = {cid: [str(cid)] for cid in communities}
579
- member_counts = {cid: len(members) for cid, members in communities.items()}
580
- to_html(meta, meta_communities, 'graphify-out/graph.html', community_labels=labels or None, member_counts=member_counts)
581
- print(f'graph.html written (aggregated: {meta.number_of_nodes()} community nodes, {meta.number_of_edges()} cross-community edges)')
582
- print('Tip: run with --obsidian for full node-level detail.')
583
- else:
584
- print('Single community — aggregated view not useful. Skipping graph.html.')
585
- else:
586
- to_html(G, communities, 'graphify-out/graph.html', community_labels=labels or None)
587
- print('graph.html written - open in any browser, no server needed')
588
- "
589
- ```
590
-
591
- ### Step 6b - Wiki (only if --wiki flag)
592
-
593
- **Only run this step if `--wiki` was explicitly given in the original command.**
594
-
595
- Run this before Step 9 (cleanup) so `graphify-out/.graphify_labels.json` is still available.
596
-
597
- ```bash
598
- $(cat graphify-out/.graphify_python) -c "
599
- import json
600
- from graphify.build import build_from_json
601
- from graphify.wiki import to_wiki
602
- from graphify.analyze import god_nodes
603
- from pathlib import Path
604
-
605
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
606
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
607
- labels_raw = json.loads(Path('graphify-out/.graphify_labels.json').read_text()) if Path('graphify-out/.graphify_labels.json').exists() else {}
608
-
609
- G = build_from_json(extraction)
610
- communities = {int(k): v for k, v in analysis['communities'].items()}
611
- cohesion = {int(k): v for k, v in analysis['cohesion'].items()}
612
- labels = {int(k): v for k, v in labels_raw.items()}
613
- gods = god_nodes(G)
614
-
615
- n = to_wiki(G, communities, 'graphify-out/wiki', community_labels=labels or None, cohesion=cohesion, god_nodes_data=gods)
616
- print(f'Wiki: {n} articles written to graphify-out/wiki/')
617
- print(' graphify-out/wiki/index.md -> agent entry point')
618
- "
619
- ```
620
-
621
- ### Step 7 - Neo4j export (only if --neo4j or --neo4j-push flag)
622
-
623
- **If `--neo4j`** - generate a Cypher file for manual import:
624
-
625
- ```bash
626
- $(cat graphify-out/.graphify_python) -c "
627
- import sys, json
628
- from graphify.build import build_from_json
629
- from graphify.export import to_cypher
630
- from pathlib import Path
631
-
632
- G = build_from_json(json.loads(Path('graphify-out/.graphify_extract.json').read_text()))
633
- to_cypher(G, 'graphify-out/cypher.txt')
634
- print('cypher.txt written - import with: cypher-shell < graphify-out/cypher.txt')
635
- "
636
- ```
637
-
638
- **If `--neo4j-push <uri>`** - push directly to a running Neo4j instance. Ask the user for credentials if not provided:
639
-
640
- ```bash
641
- $(cat graphify-out/.graphify_python) -c "
642
- import sys, json
643
- from graphify.build import build_from_json
644
- from graphify.cluster import cluster
645
- from graphify.export import push_to_neo4j
646
- from pathlib import Path
647
-
648
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
649
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
650
- G = build_from_json(extraction)
651
- communities = {int(k): v for k, v in analysis['communities'].items()}
652
-
653
- result = push_to_neo4j(G, uri='NEO4J_URI', user='NEO4J_USER', password='NEO4J_PASSWORD', communities=communities)
654
- print(f'Pushed to Neo4j: {result[\"nodes\"]} nodes, {result[\"edges\"]} edges')
655
- "
656
- ```
657
-
658
- Replace `NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD` with actual values. Default URI is `bolt://localhost:7687`, default user is `neo4j`. Uses MERGE - safe to re-run without creating duplicates.
659
-
660
- ### Step 7b - SVG export (only if --svg flag)
661
-
662
- ```bash
663
- $(cat graphify-out/.graphify_python) -c "
664
- import sys, json
665
- from graphify.build import build_from_json
666
- from graphify.export import to_svg
667
- from pathlib import Path
668
-
669
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
670
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
671
- labels_raw = json.loads(Path('graphify-out/.graphify_labels.json').read_text()) if Path('graphify-out/.graphify_labels.json').exists() else {}
672
-
673
- G = build_from_json(extraction)
674
- communities = {int(k): v for k, v in analysis['communities'].items()}
675
- labels = {int(k): v for k, v in labels_raw.items()}
676
-
677
- to_svg(G, communities, 'graphify-out/graph.svg', community_labels=labels or None)
678
- print('graph.svg written - embeds in Obsidian, Notion, GitHub READMEs')
679
- "
680
- ```
681
-
682
- ### Step 7c - GraphML export (only if --graphml flag)
683
-
684
- ```bash
685
- $(cat graphify-out/.graphify_python) -c "
686
- import json
687
- from graphify.build import build_from_json
688
- from graphify.export import to_graphml
689
- from pathlib import Path
690
-
691
- extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
692
- analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text())
693
-
694
- G = build_from_json(extraction)
695
- communities = {int(k): v for k, v in analysis['communities'].items()}
696
-
697
- to_graphml(G, communities, 'graphify-out/graph.graphml')
698
- print('graph.graphml written - open in Gephi, yEd, or any GraphML tool')
699
- "
700
- ```
701
-
702
- ### Step 7d - MCP server (only if --mcp flag)
703
-
704
- ```bash
705
- python3 -m graphify.serve graphify-out/graph.json
706
- ```
707
-
708
- This starts a stdio MCP server that exposes tools: `query_graph`, `get_node`, `get_neighbors`, `get_community`, `god_nodes`, `graph_stats`, `shortest_path`. Add to Claude Desktop or any MCP-compatible agent orchestrator so other agents can query the graph live.
709
-
710
- To configure in Claude Desktop, add to `claude_desktop_config.json`:
711
- ```json
712
- {
713
- "mcpServers": {
714
- "graphify": {
715
- "command": "python3",
716
- "args": ["-m", "graphify.serve", "/absolute/path/to/graphify-out/graph.json"]
717
- }
718
- }
719
- }
720
- ```
721
-
722
- ### Step 8 - Token reduction benchmark (only if total_words > 5000)
723
-
724
- If `total_words` from `graphify-out/.graphify_detect.json` is greater than 5,000, run:
725
-
726
- ```bash
727
- $(cat graphify-out/.graphify_python) -c "
728
- import json
729
- from graphify.benchmark import run_benchmark, print_benchmark
730
- from pathlib import Path
731
-
732
- detection = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
733
- result = run_benchmark('graphify-out/graph.json', corpus_words=detection['total_words'])
734
- print_benchmark(result)
735
- "
736
- ```
737
-
738
- Print the output directly in chat. If `total_words <= 5000`, skip silently - the graph value is structural clarity, not token compression, for small corpora.
739
-
740
- ---
741
-
742
- ### Step 9 - Save manifest, update cost tracker, clean up, and report
743
-
744
- ```bash
745
- $(cat graphify-out/.graphify_python) -c "
746
- import json
747
- from pathlib import Path
748
- from datetime import datetime, timezone
749
- from graphify.detect import save_manifest
750
-
751
- # Save manifest for --update
752
- detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
753
- save_manifest(detect['files'])
754
-
755
- # Update cumulative cost tracker
756
- extract = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
757
- input_tok = extract.get('input_tokens', 0)
758
- output_tok = extract.get('output_tokens', 0)
759
-
760
- cost_path = Path('graphify-out/cost.json')
761
- if cost_path.exists():
762
- cost = json.loads(cost_path.read_text())
763
- else:
764
- cost = {'runs': [], 'total_input_tokens': 0, 'total_output_tokens': 0}
765
-
766
- cost['runs'].append({
767
- 'date': datetime.now(timezone.utc).isoformat(),
768
- 'input_tokens': input_tok,
769
- 'output_tokens': output_tok,
770
- 'files': detect.get('total_files', 0),
771
- })
772
- cost['total_input_tokens'] += input_tok
773
- cost['total_output_tokens'] += output_tok
774
- cost_path.write_text(json.dumps(cost, indent=2))
775
-
776
- print(f'This run: {input_tok:,} input tokens, {output_tok:,} output tokens')
777
- print(f'All time: {cost[\"total_input_tokens\"]:,} input, {cost[\"total_output_tokens\"]:,} output ({len(cost[\"runs\"])} runs)')
778
- "
779
- rm -f graphify-out/.graphify_detect.json graphify-out/.graphify_extract.json graphify-out/.graphify_ast.json graphify-out/.graphify_semantic.json graphify-out/.graphify_analysis.json graphify-out/.graphify_labels.json graphify-out/.graphify_chunk_*.json
780
- rm -f graphify-out/.needs_update 2>/dev/null || true
781
- ```
782
-
783
- Tell the user (omit the obsidian line unless --obsidian was given):
784
- ```
785
- Graph complete. Outputs in PATH_TO_DIR/graphify-out/
786
-
787
- graph.html - interactive graph, open in browser
788
- GRAPH_REPORT.md - audit report
789
- graph.json - raw graph data
790
- obsidian/ - Obsidian vault (only if --obsidian was given)
791
- ```
792
-
793
- If graphify saved you time, consider supporting it: https://github.com/sponsors/safishamsi
794
-
795
- Replace PATH_TO_DIR with the actual absolute path of the directory that was processed.
796
-
797
- Then paste these sections from GRAPH_REPORT.md directly into the chat:
798
- - God Nodes
799
- - Surprising Connections
800
- - Suggested Questions
801
-
802
- Do NOT paste the full report - just those three sections. Keep it concise.
803
-
804
- Then immediately offer to explore. Pick the single most interesting suggested question from the report - the one that crosses the most community boundaries or has the most surprising bridge node - and ask:
805
-
806
- > "The most interesting question this graph can answer: **[question]**. Want me to trace it?"
807
-
808
- If the user says yes, run `/graphify query "[question]"` on the graph and walk them through the answer using the graph structure - which nodes connect, which community boundaries get crossed, what the path reveals. Keep going as long as they want to explore. Each answer should end with a natural follow-up ("this connects to X - want to go deeper?") so the session feels like navigation, not a one-shot report.
809
-
810
- The graph is the map. Your job after the pipeline is to be the guide.
811
-
812
- ---
813
-
814
- ## For --update (incremental re-extraction)
815
-
816
- Use when you've added or modified files since the last run. Only re-extracts changed files - saves tokens and time.
817
-
818
- ```bash
819
- $(cat graphify-out/.graphify_python) -c "
820
- import sys, json
821
- from graphify.detect import detect_incremental, save_manifest
822
- from pathlib import Path
823
-
824
- result = detect_incremental(Path('INPUT_PATH'))
825
- new_total = result.get('new_total', 0)
826
- print(json.dumps(result, indent=2))
827
- Path('graphify-out/.graphify_incremental.json').write_text(json.dumps(result))
828
- if new_total == 0:
829
- print('No files changed since last run. Nothing to update.')
830
- raise SystemExit(0)
831
- print(f'{new_total} new/changed file(s) to re-extract.')
832
- "
833
- ```
834
-
835
- If new files exist, first check whether all changed files are code files:
836
-
837
- ```bash
838
- $(cat graphify-out/.graphify_python) -c "
839
- import json
840
- from pathlib import Path
841
-
842
- result = json.loads(open('graphify-out/.graphify_incremental.json').read()) if Path('graphify-out/.graphify_incremental.json').exists() else {}
843
- code_exts = {'.py','.ts','.js','.go','.rs','.java','.cpp','.c','.rb','.swift','.kt','.cs','.scala','.php','.cc','.cxx','.hpp','.h','.kts'}
844
- new_files = result.get('new_files', {})
845
- all_changed = [f for files in new_files.values() for f in files]
846
- code_only = all(Path(f).suffix.lower() in code_exts for f in all_changed)
847
- print('code_only:', code_only)
848
- "
849
- ```
850
-
851
- If `code_only` is True: print `[graphify update] Code-only changes detected - skipping semantic extraction (no LLM needed)`, run only Step 3A (AST) on the changed files, skip Step 3B entirely (no subagents), then go straight to merge and Steps 4–8.
852
-
853
- If `code_only` is False (any changed file is a doc/paper/image): run the full Steps 3A–3C pipeline as normal.
854
-
855
- Then:
856
-
857
- ```bash
858
- $(cat graphify-out/.graphify_python) -c "
859
- import sys, json
860
- from graphify.build import build_from_json
861
- from graphify.export import to_json
862
- from networkx.readwrite import json_graph
863
- import networkx as nx
864
- from pathlib import Path
865
-
866
- # Load existing graph
867
- existing_data = json.loads(Path('graphify-out/graph.json').read_text())
868
- G_existing = json_graph.node_link_graph(existing_data, edges='links')
869
-
870
- # Load new extraction
871
- new_extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
872
- G_new = build_from_json(new_extraction)
873
-
874
- # Merge: new nodes/edges into existing graph
875
- G_existing.update(G_new)
876
- print(f'Merged: {G_existing.number_of_nodes()} nodes, {G_existing.number_of_edges()} edges')
877
- "
878
- ```
879
-
880
- Then run Steps 4–8 on the merged graph as normal.
881
-
882
- After Step 4, show the graph diff:
883
-
884
- ```bash
885
- $(cat graphify-out/.graphify_python) -c "
886
- import json
887
- from graphify.analyze import graph_diff
888
- from graphify.build import build_from_json
889
- from networkx.readwrite import json_graph
890
- import networkx as nx
891
- from pathlib import Path
892
-
893
- # Load old graph (before update) from backup written before merge
894
- old_data = json.loads(Path('graphify-out/.graphify_old.json').read_text()) if Path('graphify-out/.graphify_old.json').exists() else None
895
- new_extract = json.loads(Path('graphify-out/.graphify_extract.json').read_text())
896
- G_new = build_from_json(new_extract)
897
-
898
- if old_data:
899
- G_old = json_graph.node_link_graph(old_data, edges='links')
900
- diff = graph_diff(G_old, G_new)
901
- print(diff['summary'])
902
- if diff['new_nodes']:
903
- print('New nodes:', ', '.join(n['label'] for n in diff['new_nodes'][:5]))
904
- if diff['new_edges']:
905
- print('New edges:', len(diff['new_edges']))
906
- "
907
- ```
908
-
909
- Before the merge step, save the old graph: `cp graphify-out/graph.json graphify-out/.graphify_old.json`
910
- Clean up after: `rm -f graphify-out/.graphify_old.json`
911
-
912
- ---
913
-
914
- ## For --cluster-only
915
-
916
- Skip Steps 1–3. Load the existing graph from `graphify-out/graph.json` and re-run clustering:
917
-
918
- ```bash
919
- $(cat graphify-out/.graphify_python) -c "
920
- import sys, json
921
- from graphify.cluster import cluster, score_all
922
- from graphify.analyze import god_nodes, surprising_connections
923
- from graphify.report import generate
924
- from graphify.export import to_json
925
- from networkx.readwrite import json_graph
926
- import networkx as nx
927
- from pathlib import Path
928
-
929
- data = json.loads(Path('graphify-out/graph.json').read_text())
930
- G = json_graph.node_link_graph(data, edges='links')
931
-
932
- detection = {'total_files': 0, 'total_words': 99999, 'needs_graph': True, 'warning': None,
933
- 'files': {'code': [], 'document': [], 'paper': []}}
934
- tokens = {'input': 0, 'output': 0}
935
-
936
- communities = cluster(G)
937
- cohesion = score_all(G, communities)
938
- gods = god_nodes(G)
939
- surprises = surprising_connections(G, communities)
940
- labels = {cid: 'Community ' + str(cid) for cid in communities}
941
-
942
- report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, '.')
943
- Path('graphify-out/GRAPH_REPORT.md').write_text(report)
944
- to_json(G, communities, 'graphify-out/graph.json')
945
-
946
- analysis = {
947
- 'communities': {str(k): v for k, v in communities.items()},
948
- 'cohesion': {str(k): v for k, v in cohesion.items()},
949
- 'gods': gods,
950
- 'surprises': surprises,
951
- }
952
- Path('graphify-out/.graphify_analysis.json').write_text(json.dumps(analysis, indent=2))
953
- print(f'Re-clustered: {len(communities)} communities')
954
- "
955
- ```
956
-
957
- Then run Steps 5–9 as normal (label communities, generate viz, benchmark, clean up, report).
958
-
959
- ---
960
-
961
- ## For /graphify query
962
-
963
- Two traversal modes - choose based on the question:
964
-
965
- | Mode | Flag | Best for |
966
- |------|------|----------|
967
- | BFS (default) | _(none)_ | "What is X connected to?" - broad context, nearest neighbors first |
968
- | DFS | `--dfs` | "How does X reach Y?" - trace a specific chain or dependency path |
969
-
970
- First check the graph exists:
971
- ```bash
972
- $(cat graphify-out/.graphify_python) -c "
973
- from pathlib import Path
974
- if not Path('graphify-out/graph.json').exists():
975
- print('ERROR: No graph found. Run /graphify <path> first to build the graph.')
976
- raise SystemExit(1)
977
- "
978
- ```
979
- If it fails, stop and tell the user to run `/graphify <path>` first.
980
-
981
- Load `graphify-out/graph.json`, then:
982
-
983
- 1. Find the 1-3 nodes whose label best matches key terms in the question.
984
- 2. Run the appropriate traversal from each starting node.
985
- 3. Read the subgraph - node labels, edge relations, confidence tags, source locations.
986
- 4. Answer using **only** what the graph contains. Quote `source_location` when citing a specific fact.
987
- 5. If the graph lacks enough information, say so - do not hallucinate edges.
988
-
989
- ```bash
990
- $(cat graphify-out/.graphify_python) -c "
991
- import sys, json
992
- from networkx.readwrite import json_graph
993
- import networkx as nx
994
- from pathlib import Path
995
-
996
- data = json.loads(Path('graphify-out/graph.json').read_text())
997
- G = json_graph.node_link_graph(data, edges='links')
998
-
999
- question = 'QUESTION'
1000
- mode = 'MODE' # 'bfs' or 'dfs'
1001
- terms = [t.lower() for t in question.split() if len(t) > 3]
1002
-
1003
- # Find best-matching start nodes
1004
- scored = []
1005
- for nid, ndata in G.nodes(data=True):
1006
- label = ndata.get('label', '').lower()
1007
- score = sum(1 for t in terms if t in label)
1008
- if score > 0:
1009
- scored.append((score, nid))
1010
- scored.sort(reverse=True)
1011
- start_nodes = [nid for _, nid in scored[:3]]
1012
-
1013
- if not start_nodes:
1014
- print('No matching nodes found for query terms:', terms)
1015
- sys.exit(0)
1016
-
1017
- subgraph_nodes = set()
1018
- subgraph_edges = []
1019
-
1020
- if mode == 'dfs':
1021
- # DFS: follow one path as deep as possible before backtracking.
1022
- # Depth-limited to 6 to avoid traversing the whole graph.
1023
- visited = set()
1024
- stack = [(n, 0) for n in reversed(start_nodes)]
1025
- while stack:
1026
- node, depth = stack.pop()
1027
- if node in visited or depth > 6:
1028
- continue
1029
- visited.add(node)
1030
- subgraph_nodes.add(node)
1031
- for neighbor in G.neighbors(node):
1032
- if neighbor not in visited:
1033
- stack.append((neighbor, depth + 1))
1034
- subgraph_edges.append((node, neighbor))
1035
- else:
1036
- # BFS: explore all neighbors layer by layer up to depth 3.
1037
- frontier = set(start_nodes)
1038
- subgraph_nodes = set(start_nodes)
1039
- for _ in range(3):
1040
- next_frontier = set()
1041
- for n in frontier:
1042
- for neighbor in G.neighbors(n):
1043
- if neighbor not in subgraph_nodes:
1044
- next_frontier.add(neighbor)
1045
- subgraph_edges.append((n, neighbor))
1046
- subgraph_nodes.update(next_frontier)
1047
- frontier = next_frontier
1048
-
1049
- # Token-budget aware output: rank by relevance, cut at budget (~4 chars/token)
1050
- token_budget = BUDGET # default 2000
1051
- char_budget = token_budget * 4
1052
-
1053
- # Score each node by term overlap for ranked output
1054
- def relevance(nid):
1055
- label = G.nodes[nid].get('label', '').lower()
1056
- return sum(1 for t in terms if t in label)
1057
-
1058
- ranked_nodes = sorted(subgraph_nodes, key=relevance, reverse=True)
1059
-
1060
- lines = [f'Traversal: {mode.upper()} | Start: {[G.nodes[n].get(\"label\",n) for n in start_nodes]} | {len(subgraph_nodes)} nodes']
1061
- for nid in ranked_nodes:
1062
- d = G.nodes[nid]
1063
- lines.append(f' NODE {d.get(\"label\", nid)} [src={d.get(\"source_file\",\"\")} loc={d.get(\"source_location\",\"\")}]')
1064
- for u, v in subgraph_edges:
1065
- if u in subgraph_nodes and v in subgraph_nodes:
1066
- _raw = G[u][v]; d = next(iter(_raw.values()), {}) if isinstance(G, nx.MultiGraph) else _raw
1067
- lines.append(f' EDGE {G.nodes[u].get(\"label\",u)} --{d.get(\"relation\",\"\")} [{d.get(\"confidence\",\"\")}]--> {G.nodes[v].get(\"label\",v)}')
1068
-
1069
- output = '\n'.join(lines)
1070
- if len(output) > char_budget:
1071
- output = output[:char_budget] + f'\n... (truncated at ~{token_budget} token budget - use --budget N for more)'
1072
- print(output)
1073
- "
1074
- ```
1075
-
1076
- Replace `QUESTION` with the user's actual question, `MODE` with `bfs` or `dfs`, and `BUDGET` with the token budget (default `2000`, or whatever `--budget N` specifies). Then answer based on the subgraph output above.
1077
-
1078
- After writing the answer, save it back into the graph so it improves future queries:
1079
-
1080
- ```bash
1081
- $(cat graphify-out/.graphify_python) -m graphify save-result --question "QUESTION" --answer "ANSWER" --type query --nodes NODE1 NODE2
1082
- ```
1083
-
1084
- Replace `QUESTION` with the question, `ANSWER` with your full answer text, `SOURCE_NODES` with the list of node labels you cited. This closes the feedback loop: the next `--update` will extract this Q&A as a node in the graph.
1085
-
1086
- ---
1087
-
1088
- ## For /graphify path
1089
-
1090
- Find the shortest path between two named concepts in the graph.
1091
-
1092
- First check the graph exists:
1093
- ```bash
1094
- $(cat graphify-out/.graphify_python) -c "
1095
- from pathlib import Path
1096
- if not Path('graphify-out/graph.json').exists():
1097
- print('ERROR: No graph found. Run /graphify <path> first to build the graph.')
1098
- raise SystemExit(1)
1099
- "
1100
- ```
1101
- If it fails, stop and tell the user to run `/graphify <path>` first.
1102
-
1103
- ```bash
1104
- $(cat graphify-out/.graphify_python) -c "
1105
- import json, sys
1106
- import networkx as nx
1107
- from networkx.readwrite import json_graph
1108
- from pathlib import Path
1109
-
1110
- data = json.loads(Path('graphify-out/graph.json').read_text())
1111
- G = json_graph.node_link_graph(data, edges='links')
1112
-
1113
- a_term = 'NODE_A'
1114
- b_term = 'NODE_B'
1115
-
1116
- def find_node(term):
1117
- term = term.lower()
1118
- scored = sorted(
1119
- [(sum(1 for w in term.split() if w in G.nodes[n].get('label','').lower()), n)
1120
- for n in G.nodes()],
1121
- reverse=True
1122
- )
1123
- return scored[0][1] if scored and scored[0][0] > 0 else None
1124
-
1125
- src = find_node(a_term)
1126
- tgt = find_node(b_term)
1127
-
1128
- if not src or not tgt:
1129
- print(f'Could not find nodes matching: {a_term!r} or {b_term!r}')
1130
- sys.exit(0)
1131
-
1132
- try:
1133
- path = nx.shortest_path(G, src, tgt)
1134
- print(f'Shortest path ({len(path)-1} hops):')
1135
- for i, nid in enumerate(path):
1136
- label = G.nodes[nid].get('label', nid)
1137
- if i < len(path) - 1:
1138
- _raw = G[nid][path[i+1]]; edge = next(iter(_raw.values()), {}) if isinstance(G, nx.MultiGraph) else _raw
1139
- rel = edge.get('relation', '')
1140
- conf = edge.get('confidence', '')
1141
- print(f' {label} --{rel}--> [{conf}]')
1142
- else:
1143
- print(f' {label}')
1144
- except nx.NetworkXNoPath:
1145
- print(f'No path found between {a_term!r} and {b_term!r}')
1146
- except nx.NodeNotFound as e:
1147
- print(f'Node not found: {e}')
1148
- "
1149
- ```
1150
-
1151
- Replace `NODE_A` and `NODE_B` with the actual concept names from the user. Then explain the path in plain language - what each hop means, why it's significant.
1152
-
1153
- After writing the explanation, save it back:
1154
-
1155
- ```bash
1156
- $(cat graphify-out/.graphify_python) -m graphify save-result --question "Path from NODE_A to NODE_B" --answer "ANSWER" --type path_query --nodes NODE_A NODE_B
1157
- ```
1158
-
1159
- ---
1160
-
1161
- ## For /graphify explain
1162
-
1163
- Give a plain-language explanation of a single node - everything connected to it.
1164
-
1165
- First check the graph exists:
1166
- ```bash
1167
- $(cat graphify-out/.graphify_python) -c "
1168
- from pathlib import Path
1169
- if not Path('graphify-out/graph.json').exists():
1170
- print('ERROR: No graph found. Run /graphify <path> first to build the graph.')
1171
- raise SystemExit(1)
1172
- "
1173
- ```
1174
- If it fails, stop and tell the user to run `/graphify <path>` first.
1175
-
1176
- ```bash
1177
- $(cat graphify-out/.graphify_python) -c "
1178
- import json, sys
1179
- import networkx as nx
1180
- from networkx.readwrite import json_graph
1181
- from pathlib import Path
1182
-
1183
- data = json.loads(Path('graphify-out/graph.json').read_text())
1184
- G = json_graph.node_link_graph(data, edges='links')
1185
-
1186
- term = 'NODE_NAME'
1187
- term_lower = term.lower()
1188
-
1189
- # Find best matching node
1190
- scored = sorted(
1191
- [(sum(1 for w in term_lower.split() if w in G.nodes[n].get('label','').lower()), n)
1192
- for n in G.nodes()],
1193
- reverse=True
1194
- )
1195
- if not scored or scored[0][0] == 0:
1196
- print(f'No node matching {term!r}')
1197
- sys.exit(0)
1198
-
1199
- nid = scored[0][1]
1200
- data_n = G.nodes[nid]
1201
- print(f'NODE: {data_n.get(\"label\", nid)}')
1202
- print(f' source: {data_n.get(\"source_file\",\"unknown\")}')
1203
- print(f' type: {data_n.get(\"file_type\",\"unknown\")}')
1204
- print(f' degree: {G.degree(nid)}')
1205
- print()
1206
- print('CONNECTIONS:')
1207
- for neighbor in G.neighbors(nid):
1208
- _raw = G[nid][neighbor]; edge = next(iter(_raw.values()), {}) if isinstance(G, nx.MultiGraph) else _raw
1209
- nlabel = G.nodes[neighbor].get('label', neighbor)
1210
- rel = edge.get('relation', '')
1211
- conf = edge.get('confidence', '')
1212
- src_file = G.nodes[neighbor].get('source_file', '')
1213
- print(f' --{rel}--> {nlabel} [{conf}] ({src_file})')
1214
- "
1215
- ```
1216
-
1217
- Replace `NODE_NAME` with the concept the user asked about. Then write a 3-5 sentence explanation of what this node is, what it connects to, and why those connections are significant. Use the source locations as citations.
1218
-
1219
- After writing the explanation, save it back:
1220
-
1221
- ```bash
1222
- $(cat graphify-out/.graphify_python) -m graphify save-result --question "Explain NODE_NAME" --answer "ANSWER" --type explain --nodes NODE_NAME
1223
- ```
1224
-
1225
- ---
1226
-
1227
- ## For /graphify add
1228
-
1229
- Fetch a URL and add it to the corpus, then update the graph.
1230
-
1231
- ```bash
1232
- $(cat graphify-out/.graphify_python) -c "
1233
- import sys
1234
- from graphify.ingest import ingest
1235
- from pathlib import Path
1236
-
1237
- try:
1238
- out = ingest('URL', Path('./raw'), author='AUTHOR', contributor='CONTRIBUTOR')
1239
- print(f'Saved to {out}')
1240
- except ValueError as e:
1241
- print(f'error: {e}', file=sys.stderr)
1242
- sys.exit(1)
1243
- except RuntimeError as e:
1244
- print(f'error: {e}', file=sys.stderr)
1245
- sys.exit(1)
1246
- "
1247
- ```
1248
-
1249
- Replace `URL` with the actual URL, `AUTHOR` with the user's name if provided, `CONTRIBUTOR` likewise. If the command exits with an error, tell the user what went wrong - do not silently continue. After a successful save, automatically run the `--update` pipeline on `./raw` to merge the new file into the existing graph.
1250
-
1251
- Supported URL types (auto-detected):
1252
- - Twitter/X → fetched via oEmbed, saved as `.md` with tweet text and author
1253
- - arXiv → abstract + metadata saved as `.md`
1254
- - PDF → downloaded as `.pdf`
1255
- - Images (.png/.jpg/.webp) → downloaded, vision extraction runs on next build
1256
- - Any webpage → converted to markdown via html2text
1257
-
1258
- ---
1259
-
1260
- ## For --watch
1261
-
1262
- Start a background watcher that monitors a folder and auto-updates the graph when files change.
1263
-
1264
- ```bash
1265
- python3 -m graphify.watch INPUT_PATH --debounce 3
1266
- ```
1267
-
1268
- Replace INPUT_PATH with the folder to watch. Behavior depends on what changed:
1269
-
1270
- - **Code files only (.py, .ts, .go, etc.):** re-runs AST extraction + rebuild + cluster immediately, no LLM needed. `graph.json` and `GRAPH_REPORT.md` are updated automatically.
1271
- - **Docs, papers, or images:** writes a `graphify-out/needs_update` flag and prints a notification to run `/graphify --update` (LLM semantic re-extraction required).
1272
-
1273
- Debounce (default 3s): waits until file activity stops before triggering, so a wave of parallel agent writes doesn't trigger a rebuild per file.
1274
-
1275
- Press Ctrl+C to stop.
1276
-
1277
- For agentic workflows: run `--watch` in a background terminal. Code changes from agent waves are picked up automatically between waves. If agents are also writing docs or notes, you'll need a manual `/graphify --update` after those waves.
1278
-
1279
- ---
1280
-
1281
- ## For git commit hook
1282
-
1283
- Install a post-commit hook that auto-rebuilds the graph after every commit. No background process needed - triggers once per commit, works with any editor.
1284
-
1285
- ```bash
1286
- graphify hook install # install
1287
- graphify hook uninstall # remove
1288
- graphify hook status # check
1289
- ```
1290
-
1291
- After every `git commit`, the hook detects which code files changed (via `git diff HEAD~1`), re-runs AST extraction on those files, and rebuilds `graph.json` and `GRAPH_REPORT.md`. Doc/image changes are ignored by the hook - run `/graphify --update` manually for those.
1292
-
1293
- If a post-commit hook already exists, graphify appends to it rather than replacing it.
1294
-
1295
- ---
1296
-
1297
- ## For native CLAUDE.md integration
1298
-
1299
- Run once per project to make graphify always-on in Claude Code sessions:
1300
-
1301
- ```bash
1302
- graphify claude install
1303
- ```
1304
-
1305
- This writes a `## graphify` section to the local `CLAUDE.md` that instructs Claude to check the graph before answering codebase questions and rebuild it after code changes. No manual `/graphify` needed in future sessions.
1306
-
1307
- ```bash
1308
- graphify claude uninstall # remove the section
1309
- ```
1310
-
1311
- ---
1312
-
1313
- ## Honesty Rules
1314
-
1315
- - Never invent an edge. If unsure, use AMBIGUOUS.
1316
- - Never skip the corpus check warning.
1317
- - Always show token cost in the report.
1318
- - Never hide cohesion scores behind symbols - show the raw number.
1319
- - Never run HTML viz on a graph with more than 5,000 nodes without warning the user.