wikifier 4.1.3__tar.gz → 4.2.0__tar.gz

{wikifier-4.1.3 → wikifier-4.2.0}/MANIFEST.in

@@ -3,10 +3,6 @@ include README.md
 include LICENSE
 include CONTRIBUTING.md
 include docs/spec.md
-include docs/Basis-v0.3.md
-include docs/TRADEOFFS.md
-include docs/RELEASE_NOTES.md
-include docs/v0.4-*.md
 include skills/run.md
 include index.html
 include diagnostics.html
@@ -17,3 +13,5 @@ prune journal
 prune Logged_issues
 prune .wikifier_staging
 prune .chisel
+prune wikifier/scripts/journal
+prune wikifier/scripts/.wikifier_staging

wikifier-4.2.0/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.4
+Name: wikifier
+Version: 4.2.0
+Summary: Zero-dependency agent-to-agent codebase wiki for LLMs and AI agents. Autonomously maintained via record-change and mark-green for token-efficient lookup of files, dependencies, health, and summaries — across tiny scripts to 50k+ monorepos. Optional MCP server with rich tools for agents.
+Author-email: Aron Amos <aron@example.com>
+Maintainer: Aron Amos
+License: MIT
+Project-URL: Homepage, https://github.com/IronAdamant/wikifier
+Project-URL: Repository, https://github.com/IronAdamant/wikifier
+Project-URL: Documentation, https://github.com/IronAdamant/wikifier#readme
+Project-URL: Bug Tracker, https://github.com/IronAdamant/wikifier/issues
+Keywords: wiki,documentation,llm,agent,mcp,codebase,health-matrix,zero-dependency,shell,token-efficient,autonomous,record-change,mark-green,agent-wiki,llm-tools,dependency-graph,monorepo
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Software Development :: Version Control
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0.0; extra == "mcp"
+Dynamic: license-file
+
+# Wikifier
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI version](https://img.shields.io/pypi/v/wikifier.svg)](https://pypi.org/project/wikifier/)
+[![GitHub Stars](https://img.shields.io/github/stars/IronAdamant/wikifier?style=social)](https://github.com/IronAdamant/wikifier/stargazers)
+
+**The agent-to-agent codebase wiki — a token-efficient codebase map for AI agents. Zero dependencies, autonomous, built for LLM context windows.**
+
+Wikifier gives AI coding agents and LLMs a living, queryable map of any codebase — file health matrix, dependency graphs, per-file summaries — so they can look things up instead of re-reading full source files into their context window. From small scripts to 50,000-file monorepos, agents get fast code navigation, import analysis, and circular dependency detection without burning tokens on code they don't need right now.
+
+It is an automated library for AI agents: an agent asks for the map, gets a compact answer, and keeps the wiki current itself as it works.
+
+## Why Wikifier?
+
+Every AI coding assistant faces the same problem: context windows are finite and raw source code is expensive. Re-reading a 2,000-line file to answer "what does this module do and who imports it?" wastes tokens that could go toward actual reasoning.
+
+Wikifier solves this with a small set of generated, queryable artifacts:
+
+- **`file_health.md`** — a documentation health matrix (🟢 current / 🟡 stale / 🔴 broken) so agents know what to trust and what to fix first
+- **`library.md`** — the codebase map: a Mermaid dependency diagram, resolved import tables, circular dependency report with break recommendations, and a confidence-scored risk snapshot
+- **Per-file `*.wiki.md` notes** — short "what this file is for" summaries, maintained by the agents themselves
+- **`journal/`** — a semantic audit trail of *why* every change was made, written by `record-change`
+
+The result: an LLM looks up a 50-token summary instead of ingesting a 5,000-token file. That's the entire point — token-efficient code understanding for AI agents.
+
+## How It Works — the Agent Loop
+
+Agents keep the wiki alive through a strict but lightweight workflow (the full LLM-ready protocol lives in [`skills/run.md`](skills/run.md)):
+
+```bash
+wikifier check-changes          # what changed? (incremental, sub-second)
+# read file_health.md + pending_updates.md, prioritize 🔴 then 🟡
+# ... edit source ...
+wikifier record-change "path/file.py" "why this changed"   # mandatory — the semantic audit trail
+# ... update the file's wiki summary ...
+wikifier mark-green "path/file.py"
+wikifier update-maps            # when imports/structure changed
+```
+
+`record-change` is the heart of the system: it captures intent (the *why*), which is exactly the context the next agent session — or the next human — can't recover from a git diff alone.
+
+## Installation & Quick Start
+
+```bash
+pip install wikifier            # zero-dependency core (pure Python stdlib)
+pip install wikifier[mcp]       # optional: adds the MCP server for AI agents
+```
+
+Bootstrap any project:
+
+```bash
+cd /path/to/your/project
+wikifier init                   # creates monitored_paths.txt, excludes, and the human dashboard
+wikifier update-maps            # build the dependency graph + library.md
+wikifier check-changes          # start the loop
+wikifier health --summary       # compact machine-readable status
+```
+
+For external projects or monorepos, always pass an explicit root: `WIKIFIER_PROJECT_ROOT=/abs/path wikifier ...`
+
+## What You Get
+
+- **Fast, real dependency analysis** — Python and JavaScript/TypeScript import parsing (ES modules, CommonJS, dynamic imports, TypeScript path aliases, package.json exports, workspaces), barrel/re-export chain expansion, and per-edge confidence scoring with actionable explanations
+- **A pure-Python update pipeline** — `update-maps` parses every changed file in-process, persists a canonical import cache, computes reverse dependencies, Tarjan-based circular dependency detection, and regenerates `library.md` atomically
+- **Incremental everything** — mtime-based dirty detection plus a barrel-aware reverse index means editing one file re-analyzes only its true consumers, even in barrel-heavy monorepos
+- **Autonomous maintenance** — agents log intent with `record-change`, refresh summaries, and `mark-green`; a background `monitor`/daemon keeps the matrix fresh
+- **An optional MCP server** — 23+ Model Context Protocol tools (`get_project_status`, `get_dependencies`, `get_file_wiki`, `get_cycles`, `suggest_next_actions`, …) for Claude Code, Claude Desktop, Cursor, Cline, and any MCP-capable AI agent
+- **A clean human dashboard** — `index.html`, a static read-only viewer of the same artifacts: the dependency chart, files with plain-language descriptions, and one-click copy buttons (see below)
+- **True zero dependencies** — the core runs on the Python standard library and POSIX shell alone, so forks can layer their own libraries on a dependency-free base
+
+## Real-World Performance (v4.2.0, measured)
+
+Validated by dogfooding on Wikifier itself plus large open-source codebases — Apache Airflow, Babylon.js, LLVM, the Linux kernel, Rust, and llama_index:
+
+| Codebase | Scale | Full `update-maps` |
+|----------|-------|--------------------|
+| llama_index | 3,837 Python files | **~8.5s** (17k import edges, full map) |
+| Babylon.js | 3,905 TypeScript files, barrel-heavy | completes with full 5,389-node map |
+| Linux kernel / LLVM / Rust | 37k–54k file trees | candidate scan in 3–8s |
+
+JS/TS parsing runs at ~22ms per file; Python at ~1ms. Incremental runs after the first build take seconds. Verified by a stdlib-only test suite (`python -m unittest discover tests`).
+
+## Core Commands
+
+| Command | Purpose |
+|---------|---------|
+| `wikifier init [--target DIR]` | Bootstrap a project (templates + human dashboard) |
+| `wikifier check-changes` | Incremental change scan + health/pending update |
+| `wikifier record-change <file> "reason"` | Log the *why* (required after edits) |
+| `wikifier mark-green <file>` | Mark the wiki entry current |
+| `wikifier update-maps [--full] [--directory=src/]` | Rebuild dependency graph + library.md (pure-Python pipeline; `--sh` for the legacy shell path) |
+| `wikifier health [--summary\|--json]` | Health matrix — compact formats for agents |
+| `wikifier cycles` | Circular dependency report with break recommendations |
+| `wikifier monitor &` / `wikifier daemon start` | Background heartbeat / managed maintenance |
+
+Python library access for full power: `from wikifier import check_changes, record_change, mark_green, health, update_maps`.
+
+## MCP Server for AI Coding Agents
+
+```bash
+WIKIFIER_PROJECT_ROOT=/abs/path/to/project wikifier-mcp
+```
+
+Works with Claude Desktop, Claude Code, Cursor, Cline, and any Model Context Protocol client. Root detection priority: env var → explicit `project_root=` per call → marker walk → cwd. Setup, tool list, and client config examples: [`wikifier/mcp/README.md`](wikifier/mcp/README.md).
+
+## The Human Layer (Secondary, by Design)
+
+`wikifier init` copies a single static `index.html` into your project: a read-only dashboard showing the Mermaid code-structure chart, files with short descriptions, and a folder browser — useful for humans investigating what the agents know. Command buttons copy the exact CLI command and live-poll for results (a static page can't execute shell commands — that's the browser sandbox working as intended). The agent-facing markdown files and tools remain the single source of truth.
+
+## Intended Use: Strictly Agent-to-Agent
+
+Wikifier is a token-saving wiki layer *for agents*, and deliberately nothing more: not a general documentation generator, not an IDE plugin, not a code search engine for humans. Agents consult the map instead of re-reading sources, keep it current as they work, and create new wiki entries on the fly. The human dashboard is a window into that, not the product.
+
+## What's New in v4.2.0
+
+A full real-world dogfood sweep (this repo + RecipeLab + 8 large OSS projects) followed by a fix pass for everything it surfaced:
+
+- `update-maps` now defaults to the **pure-Python pipeline**: every changed file actually parsed, canonical cache schema, cycles/ACS/reverse-deps computed, `library.md` regenerated atomically
+- **43× faster JS/TS parsing** (barrel-cache persistence batched per run instead of per chain)
+- Fixed a POSIX lock self-deadlock in the library workflow calls (the likely cause of historical MCP timeouts), barrel-churn invalidation, static-import flag pollution, exclude-pattern globs, `health --summary/--json`, and several shell-path crashes — one of which could destroy an existing `library.md`
+- **New stdlib-only test suite** (28 tests) — the project's first formal regression net
+
+Full details: [`CHANGELOG.md`](CHANGELOG.md) and `Findings/2026-06-10-Dogfood-Refactor-Validation.md`.
+
+## Links
+
+- **GitHub**: https://github.com/IronAdamant/wikifier
+- **PyPI**: https://pypi.org/project/wikifier/
+- **Agent Protocol**: [`skills/run.md`](skills/run.md) — read this first as an agent
+- **MCP Setup**: [`wikifier/mcp/README.md`](wikifier/mcp/README.md)
+- **Evidence**: `Findings/` — real-world dogfood reports and metrics
+
+**For AI search / agents**: Wikifier is a zero-dependency, agent-maintained codebase wiki and dependency graph generator that gives LLMs and AI coding agents token-efficient codebase maps — health matrix, Mermaid dependency diagrams, import analysis, circular dependency detection, and an optional MCP server — with autonomous `record-change` / `mark-green` updates, validated on monorepos up to 50k+ files.

wikifier-4.2.0/README.md

@@ -0,0 +1,133 @@
+# Wikifier
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI version](https://img.shields.io/pypi/v/wikifier.svg)](https://pypi.org/project/wikifier/)
+[![GitHub Stars](https://img.shields.io/github/stars/IronAdamant/wikifier?style=social)](https://github.com/IronAdamant/wikifier/stargazers)
+
+**The agent-to-agent codebase wiki — a token-efficient codebase map for AI agents. Zero dependencies, autonomous, built for LLM context windows.**
+
+Wikifier gives AI coding agents and LLMs a living, queryable map of any codebase — file health matrix, dependency graphs, per-file summaries — so they can look things up instead of re-reading full source files into their context window. From small scripts to 50,000-file monorepos, agents get fast code navigation, import analysis, and circular dependency detection without burning tokens on code they don't need right now.
+
+It is an automated library for AI agents: an agent asks for the map, gets a compact answer, and keeps the wiki current itself as it works.
+
+## Why Wikifier?
+
+Every AI coding assistant faces the same problem: context windows are finite and raw source code is expensive. Re-reading a 2,000-line file to answer "what does this module do and who imports it?" wastes tokens that could go toward actual reasoning.
+
+Wikifier solves this with a small set of generated, queryable artifacts:
+
+- **`file_health.md`** — a documentation health matrix (🟢 current / 🟡 stale / 🔴 broken) so agents know what to trust and what to fix first
+- **`library.md`** — the codebase map: a Mermaid dependency diagram, resolved import tables, circular dependency report with break recommendations, and a confidence-scored risk snapshot
+- **Per-file `*.wiki.md` notes** — short "what this file is for" summaries, maintained by the agents themselves
+- **`journal/`** — a semantic audit trail of *why* every change was made, written by `record-change`
+
+The result: an LLM looks up a 50-token summary instead of ingesting a 5,000-token file. That's the entire point — token-efficient code understanding for AI agents.
+
+## How It Works — the Agent Loop
+
+Agents keep the wiki alive through a strict but lightweight workflow (the full LLM-ready protocol lives in [`skills/run.md`](skills/run.md)):
+
+```bash
+wikifier check-changes          # what changed? (incremental, sub-second)
+# read file_health.md + pending_updates.md, prioritize 🔴 then 🟡
+# ... edit source ...
+wikifier record-change "path/file.py" "why this changed"   # mandatory — the semantic audit trail
+# ... update the file's wiki summary ...
+wikifier mark-green "path/file.py"
+wikifier update-maps            # when imports/structure changed
+```
+
+`record-change` is the heart of the system: it captures intent (the *why*), which is exactly the context the next agent session — or the next human — can't recover from a git diff alone.
+
+## Installation & Quick Start
+
+```bash
+pip install wikifier            # zero-dependency core (pure Python stdlib)
+pip install wikifier[mcp]       # optional: adds the MCP server for AI agents
+```
+
+Bootstrap any project:
+
+```bash
+cd /path/to/your/project
+wikifier init                   # creates monitored_paths.txt, excludes, and the human dashboard
+wikifier update-maps            # build the dependency graph + library.md
+wikifier check-changes          # start the loop
+wikifier health --summary       # compact machine-readable status
+```
+
+For external projects or monorepos, always pass an explicit root: `WIKIFIER_PROJECT_ROOT=/abs/path wikifier ...`
+
+## What You Get
+
+- **Fast, real dependency analysis** — Python and JavaScript/TypeScript import parsing (ES modules, CommonJS, dynamic imports, TypeScript path aliases, package.json exports, workspaces), barrel/re-export chain expansion, and per-edge confidence scoring with actionable explanations
+- **A pure-Python update pipeline** — `update-maps` parses every changed file in-process, persists a canonical import cache, computes reverse dependencies, Tarjan-based circular dependency detection, and regenerates `library.md` atomically
+- **Incremental everything** — mtime-based dirty detection plus a barrel-aware reverse index means editing one file re-analyzes only its true consumers, even in barrel-heavy monorepos
+- **Autonomous maintenance** — agents log intent with `record-change`, refresh summaries, and `mark-green`; a background `monitor`/daemon keeps the matrix fresh
+- **An optional MCP server** — 23+ Model Context Protocol tools (`get_project_status`, `get_dependencies`, `get_file_wiki`, `get_cycles`, `suggest_next_actions`, …) for Claude Code, Claude Desktop, Cursor, Cline, and any MCP-capable AI agent
+- **A clean human dashboard** — `index.html`, a static read-only viewer of the same artifacts: the dependency chart, files with plain-language descriptions, and one-click copy buttons (see below)
+- **True zero dependencies** — the core runs on the Python standard library and POSIX shell alone, so forks can layer their own libraries on a dependency-free base
+
+## Real-World Performance (v4.2.0, measured)
+
+Validated by dogfooding on Wikifier itself plus large open-source codebases — Apache Airflow, Babylon.js, LLVM, the Linux kernel, Rust, and llama_index:
+
+| Codebase | Scale | Full `update-maps` |
+|----------|-------|--------------------|
+| llama_index | 3,837 Python files | **~8.5s** (17k import edges, full map) |
+| Babylon.js | 3,905 TypeScript files, barrel-heavy | completes with full 5,389-node map |
+| Linux kernel / LLVM / Rust | 37k–54k file trees | candidate scan in 3–8s |
+
+JS/TS parsing runs at ~22ms per file; Python at ~1ms. Incremental runs after the first build take seconds. Verified by a stdlib-only test suite (`python -m unittest discover tests`).
+
+## Core Commands
+
+| Command | Purpose |
+|---------|---------|
+| `wikifier init [--target DIR]` | Bootstrap a project (templates + human dashboard) |
+| `wikifier check-changes` | Incremental change scan + health/pending update |
+| `wikifier record-change <file> "reason"` | Log the *why* (required after edits) |
+| `wikifier mark-green <file>` | Mark the wiki entry current |
+| `wikifier update-maps [--full] [--directory=src/]` | Rebuild dependency graph + library.md (pure-Python pipeline; `--sh` for the legacy shell path) |
+| `wikifier health [--summary\|--json]` | Health matrix — compact formats for agents |
+| `wikifier cycles` | Circular dependency report with break recommendations |
+| `wikifier monitor &` / `wikifier daemon start` | Background heartbeat / managed maintenance |
+
+Python library access for full power: `from wikifier import check_changes, record_change, mark_green, health, update_maps`.
+
+## MCP Server for AI Coding Agents
+
+```bash
+WIKIFIER_PROJECT_ROOT=/abs/path/to/project wikifier-mcp
+```
+
+Works with Claude Desktop, Claude Code, Cursor, Cline, and any Model Context Protocol client. Root detection priority: env var → explicit `project_root=` per call → marker walk → cwd. Setup, tool list, and client config examples: [`wikifier/mcp/README.md`](wikifier/mcp/README.md).
+
+## The Human Layer (Secondary, by Design)
+
+`wikifier init` copies a single static `index.html` into your project: a read-only dashboard showing the Mermaid code-structure chart, files with short descriptions, and a folder browser — useful for humans investigating what the agents know. Command buttons copy the exact CLI command and live-poll for results (a static page can't execute shell commands — that's the browser sandbox working as intended). The agent-facing markdown files and tools remain the single source of truth.
+
+## Intended Use: Strictly Agent-to-Agent
+
+Wikifier is a token-saving wiki layer *for agents*, and deliberately nothing more: not a general documentation generator, not an IDE plugin, not a code search engine for humans. Agents consult the map instead of re-reading sources, keep it current as they work, and create new wiki entries on the fly. The human dashboard is a window into that, not the product.
+
+## What's New in v4.2.0
+
+A full real-world dogfood sweep (this repo + RecipeLab + 8 large OSS projects) followed by a fix pass for everything it surfaced:
+
+- `update-maps` now defaults to the **pure-Python pipeline**: every changed file actually parsed, canonical cache schema, cycles/ACS/reverse-deps computed, `library.md` regenerated atomically
+- **43× faster JS/TS parsing** (barrel-cache persistence batched per run instead of per chain)
+- Fixed a POSIX lock self-deadlock in the library workflow calls (the likely cause of historical MCP timeouts), barrel-churn invalidation, static-import flag pollution, exclude-pattern globs, `health --summary/--json`, and several shell-path crashes — one of which could destroy an existing `library.md`
+- **New stdlib-only test suite** (28 tests) — the project's first formal regression net
+
+Full details: [`CHANGELOG.md`](CHANGELOG.md) and `Findings/2026-06-10-Dogfood-Refactor-Validation.md`.
+
+## Links
+
+- **GitHub**: https://github.com/IronAdamant/wikifier
+- **PyPI**: https://pypi.org/project/wikifier/
+- **Agent Protocol**: [`skills/run.md`](skills/run.md) — read this first as an agent
+- **MCP Setup**: [`wikifier/mcp/README.md`](wikifier/mcp/README.md)
+- **Evidence**: `Findings/` — real-world dogfood reports and metrics
+
+**For AI search / agents**: Wikifier is a zero-dependency, agent-maintained codebase wiki and dependency graph generator that gives LLMs and AI coding agents token-efficient codebase maps — health matrix, Mermaid dependency diagrams, import analysis, circular dependency detection, and an optional MCP server — with autonomous `record-change` / `mark-green` updates, validated on monorepos up to 50k+ files.

{wikifier-4.1.3 → wikifier-4.2.0}/index.html

@@ -125,15 +125,21 @@
                 </div>
             </div>
 
-            <!-- Quick actions: buttons for the main commands (copy to clipboard so user can run in terminal) -->
+            <!-- Quick actions: buttons for the main commands.
+                 Why copy + wait, not direct execute?
+                 This page is a pure static client-side viewer (no server, no privileged code, zero runtime deps).
+                 Browser JS cannot execute host shell commands (security sandbox + design choice for portability and simplicity).
+                 The buttons prepare the exact command and immediately start live monitoring so the results appear automatically once you run the command in your terminal.
+                 This is the intended model: human view on top of agent-executed tools.
+            -->
             <div class="mb-4 px-1">
-              <div class="text-[10px] text-slate-400 mb-1">Quick actions — click to copy the command, paste &amp; run in your terminal (this folder), then refresh the page</div>
+              <div class="text-[10px] text-slate-400 mb-1">Quick actions — click to copy the command + start live auto-update. Paste &amp; run in your terminal (this folder); the page will detect changes and refresh the view automatically.</div>
               <div class="flex flex-wrap gap-2">
-                <button onclick="copyCommandForRun('check-changes')" 
+                <button onclick="copyAndWaitForUpdate('check-changes')" 
                         class="px-3 py-1.5 text-sm rounded-xl bg-slate-800 hover:bg-slate-700 border border-slate-700 flex items-center gap-x-2">
                   <span>Check for changes</span>
                 </button>
-                <button onclick="copyCommandForRun('update-maps')" 
+                <button onclick="copyAndWaitForUpdate('update-maps')" 
                         class="px-3 py-1.5 text-sm rounded-xl bg-emerald-800 hover:bg-emerald-700 border border-emerald-700 text-emerald-300 flex items-center gap-x-2">
                   <span>Rebuild structure map</span>
                 </button>
@@ -151,7 +157,7 @@
                         <span class="font-semibold text-lg">Code structure</span>
                         <span class="ml-2 text-xs text-slate-500">(imports &amp; dependencies — rebuilt from your code)</span>
                     </div>
-                    <button onclick="runCommand('update-maps')" 
+                    <button onclick="copyAndWaitForUpdate('update-maps')" 
                             class="text-xs px-3 py-1.5 rounded-xl bg-emerald-900/30 hover:bg-emerald-900/50 text-emerald-400 border border-emerald-800">Rebuild tree</button>
                 </div>
                 <div id="mermaid-container" class="bg-slate-950 border border-slate-800 rounded-2xl p-5 min-h-[340px]">
@@ -180,7 +186,7 @@
                                oninput="filterFilesList()" onkeydown="if(event.key==='Enter')filterFilesList()">
                         <button onclick="filterFilesList()" 
                                 class="text-xs px-3 py-1.5 rounded-xl bg-slate-800 border border-slate-700 hover:bg-slate-700">Filter</button>
-                        <button onclick="runCommand('check-changes')" 
+                        <button onclick="copyAndWaitForUpdate('check-changes')" 
                                 class="text-xs px-3 py-1.5 rounded-xl bg-emerald-900/30 hover:bg-emerald-900/50 text-emerald-400 border border-emerald-800">
                             Update data
                         </button>
@@ -361,7 +367,7 @@
                 listEl.innerHTML = `<div class="px-5 py-8 text-center">
                     <div class="text-sm text-slate-400">No files in the wiki yet.</div>
                     <div class="mt-3">
-                        <button onclick="copyCommandForRun('check-changes &amp;&amp; update-maps', 'wikifier check-changes &amp;&amp; wikifier update-maps')" 
+                        <button onclick="copyAndWaitForUpdate('check-changes &amp;&amp; update-maps', 'wikifier check-changes &amp;&amp; wikifier update-maps')" 
                                 class="px-4 py-2 bg-emerald-600 hover:bg-emerald-500 text-white rounded-xl text-sm font-medium">
                             📋 Copy first-time setup command
                         </button>
@@ -550,7 +556,7 @@
                     <div class="font-semibold mb-1">No structure map yet</div>
                     <div class="text-slate-400">This project hasn't had its import/dependency tree generated yet.</div>
                     <div class="mt-3">
-                        <button onclick="copyCommandForRun('update-maps')" 
+                        <button onclick="copyAndWaitForUpdate('update-maps')" 
                                 class="px-4 py-2 bg-emerald-600 hover:bg-emerald-500 text-white rounded-xl text-sm font-medium w-full sm:w-auto">
                             📋 Copy command &amp; run wikifier update-maps (first time)
                         </button>
@@ -559,7 +565,7 @@
                 </div>`;
 
                 // "Automatically on the first run": if this is the very first time the user sees the empty map state in this browser session,
-                // auto-copy the command for them so it feels automatic, with clear feedback.
+                // auto-copy the command AND activate live wait/poll mode so once they run it, the tree appears automatically.
                 if (!sessionStorage.getItem('wikifier_first_map_auto_copied')) {
                     sessionStorage.setItem('wikifier_first_map_auto_copied', '1');
                     setTimeout(() => {
@@ -570,6 +576,9 @@
                             note.textContent = '✓ update-maps command auto-copied for first run — paste it in your terminal now.';
                             const emptyBox = container.querySelector('div');
                             if (emptyBox) emptyBox.appendChild(note);
+                            // Also start the live wait/poll immediately for the "auto" feel
+                            // (user still needs to execute the command in terminal)
+                            copyAndWaitForUpdate('update-maps', 'wikifier update-maps'); // this will re-copy but mainly starts the banner + poll
                         }).catch(() => {});
                     }, 300);
                 }
@@ -686,6 +695,75 @@ Open index.html in the project for the live visual version.
             if (cmd.includes('monitor')) setTimeout(() => loadMonitorStatus && loadMonitorStatus(true), 600);
         }
 
+        // Enhanced: copy + enter "live waiting" mode with aggressive polling until data appears.
+        // This makes buttons feel like they "trigger" an update (user still runs the copied command in terminal).
+        let waitPollInterval = null;
+        function copyAndWaitForUpdate(cmd, fullText) {
+            const full = fullText || `wikifier ${cmd}`;
+            navigator.clipboard.writeText(full).then(() => {
+                // Show waiting banner
+                let banner = document.getElementById('wait-banner');
+                if (!banner) {
+                    banner = document.createElement('div');
+                    banner.id = 'wait-banner';
+                    banner.className = `fixed top-4 left-1/2 -translate-x-1/2 bg-blue-900 border border-blue-700 px-4 py-2 rounded-2xl text-sm z-50 flex items-center gap-3 shadow-lg`;
+                    document.body.appendChild(banner);
+                }
+                banner.innerHTML = `
+                    <span>Waiting for <code class="bg-black/40 px-1 rounded">${full}</code> to produce data... (auto-polling)</span>
+                    <button onclick="stopWaitPoll(); this.closest('.fixed').remove(); window.location.reload()" class="text-xs underline">I ran it — refresh now</button>
+                `;
+
+                // Clear any old poll
+                if (waitPollInterval) clearInterval(waitPollInterval);
+
+                let attempts = 0;
+                const maxAttempts = 40; // ~2 minutes at 3s intervals
+                waitPollInterval = setInterval(() => {
+                    attempts++;
+                    // Reload the relevant views
+                    if (cmd.includes('update') || cmd.includes('map')) {
+                        loadMermaid();
+                    }
+                    if (cmd.includes('check') || cmd.includes('health')) {
+                        loadHealthMatrix();
+                    }
+                    loadMonitorStatus();
+
+                    // Check if data appeared
+                    const hasMap = !!window.libraryMdCache && window.libraryMdCache.includes('```mermaid');
+                    const hasFiles = window.healthData && window.healthData.length > 0;
+
+                    if ((cmd.includes('update') && hasMap) || (cmd.includes('check') && hasFiles) || (hasMap && hasFiles)) {
+                        stopWaitPoll();
+                        if (banner.parentNode) banner.parentNode.removeChild(banner);
+                        // Show success toast
+                        const success = document.createElement('div');
+                        success.className = `fixed bottom-4 right-4 bg-emerald-900 border border-emerald-700 px-4 py-2 rounded-2xl text-sm z-50`;
+                        success.textContent = '✓ Data updated! View refreshed.';
+                        document.body.appendChild(success);
+                        setTimeout(() => { if (success.parentNode) success.parentNode.removeChild(success); }, 3000);
+                        // Force full reload of views
+                        loadMermaid();
+                        loadHealthMatrix();
+                    } else if (attempts >= maxAttempts) {
+                        stopWaitPoll();
+                        if (banner.parentNode) banner.parentNode.removeChild(banner);
+                    }
+                }, 3000); // aggressive 3s poll while waiting
+            }).catch(() => {
+                // fallback
+                copyCommandForRun(cmd, full);
+            });
+        }
+
+        function stopWaitPoll() {
+            if (waitPollInterval) {
+                clearInterval(waitPollInterval);
+                waitPollInterval = null;
+            }
+        }
+
         // Legacy / existing buttons still work via the improved helper
         function runCommand(cmd) {
             copyCommandForRun(cmd);

{wikifier-4.1.3 → wikifier-4.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "wikifier"
-version = "4.1.3"
+dynamic = ["version"]
 description = "Zero-dependency agent-to-agent codebase wiki for LLMs and AI agents. Autonomously maintained via record-change and mark-green for token-efficient lookup of files, dependencies, health, and summaries — across tiny scripts to 50k+ monorepos. Optional MCP server with rich tools for agents."
 readme = "README.md"
 license = {text = "MIT"}
@@ -57,10 +57,10 @@ package-dir = {"" = "."}
 
 [tool.setuptools.packages.find]
 where = ["."]
-include = ["wikifier*", "index.html", "diagnostics.html", "docs/*"]
+include = ["wikifier*"]
 
 [tool.setuptools.package-data]
-wikifier = ["scripts/*"]
+wikifier = ["scripts/*", "index.html"]
 
 [tool.setuptools.exclude-package-data]
 "*" = ["*.pyc", "__pycache__"]

{wikifier-4.1.3 → wikifier-4.2.0}/skills/run.md

@@ -1,11 +1,18 @@
-# Wikifier Agent Protocol v0.5 (M5 Update)
+# Wikifier Agent Protocol v0.6
 
 **Formerly "Wikifier Skills & Commands". This is the authoritative, versioned specification for agent behavior when using Wikifier.**
 
-**Version**: v0.5 (M5 Post-Dogfood + Hardening + Sustained Update)  
-**Date**: 2026-06-04  
-**Status**: Active. Supersedes v0.4.  
-**See also**: `README.md` (Intended Use section: strictly agent-to-agent wiki for token saving), M5-Dogfood-Assessment-Report.md, M5-Dogfood-Progress.md, and the library in wikifier/.
+**Version**: v0.6 (v4.2.0 real-pipeline + reliability update)  
+**Date**: 2026-06-10  
+**Status**: Active. Supersedes v0.5.  
+**See also**: `README.md` (Intended Use section: strictly agent-to-agent wiki for token saving), M5-Dogfood-Assessment-Report.md, Findings/2026-06-10-Dogfood-Refactor-Validation.md, and the library in wikifier/.
+
+**v0.6 migration notes (package v4.2.0)** — changes are additive or strictly-better defaults; v0.5 agent behavior keeps working:
+- `update-maps` (CLI) now runs the **pure-Python full pipeline by default**: every dirty file is parsed in-process, the canonical per-file cache is persisted, reverse deps + cycles + ACS are computed, and `library.md` is regenerated atomically. `--python-primary` is accepted but redundant; `--sh`/`--legacy-sh` selects the legacy shell path (slow fallback). Scoping is explicit via `--directory=`/`--max-files=` and reported in the result (`files_skipped`) — there are no silent caps.
+- `run_full_update`/`update_maps` returns gained additive fields: `files_parsed` (actual), `parseable_files`, `edges_persisted`, `files_skipped`, `cycles`, `library`. Tolerate additive fields as always.
+- `wikifier health --summary|--json|--format=...` now work from the CLI (previously library-only).
+- A POSIX lock self-deadlock in library `record_change`/`mark_green`/`check_changes` was fixed (locking is now re-entrant per process). MCP timeouts on those tools should no longer occur; the CLI fallback guidance below remains valid as defense in depth.
+- A stdlib-only test suite exists: `python -m unittest discover tests` (run it when working on Wikifier itself).
 
 **Core design**: Token-efficient agent-to-agent wiki. Agents use it to look up files (health matrix + file wikis + BRC + stele chunks instead of full sources), autonomously update wiki summaries, and create new ones as work progresses. **It should not be used for anything more than that.** 
 
@@ -20,20 +27,20 @@ This protocol + the Python package + MCP server form the bridge for consistent,
 **Copy this exact block into the system prompt or the very first instruction of every new LLM session that uses Wikifier:**
 
 ```
-You are now operating inside a Wikifier v0.5 managed codebase (Agent Protocol v0.5 — M5 update).
+You are now operating inside a Wikifier-managed codebase (Agent Protocol v0.6 — v4.2.0).
 
 This is strictly an agent-to-agent wiki for token saving: quick file lookup (health matrix + file wikis + BRC + stele instead of full sources), autonomous wiki summary updates, and creation of new wiki entries as you work. It should not be used for anything more.
 
 FIRST ACTIONS (mandatory, post-M5 hardened):
 1. If the Wikifier MCP server is connected, prefer its tools (get_project_status, get_file_wiki, check_changes, get_barrel_reports, suggest_next_actions, etc.). Always pass project_root= for external/user projects (e.g. RecipeLab_alt, ConsistencyHub, llvm subs).
-2. If MCP times out or is unavailable (common on BRC-heavy ~20+ yellow or 50k+ scale like alt/Consistency/llvm per M5 dogfood), fall back immediately to reliable CLI/library:
+2. If MCP times out or is unavailable (rare since v4.2.0 fixed the workflow-call deadlock and batched barrel persistence; historically seen on BRC-heavy or 50k+ scale targets), fall back immediately to reliable CLI/library:
      WIKIFIER_PROJECT_ROOT=/path/to/target python -m wikifier check-changes
      python -m wikifier health --summary   # or format=json/summary for machine use (token-efficient lookup)
      ... perform edit on source ...
      python -m wikifier record-change "path/to/file" "concise semantic reason (why, not what). Include subid if agent work."
      ... write/update the wiki summary for the file ...
      python -m wikifier mark-green "path/to/file" "Reason for Green (e.g. after autonomous agent update)."
-     if imports changed: python -m wikifier update-maps --directory=src/  # Python-primary for large
+     if imports changed: python -m wikifier update-maps [--directory=src/]  # pure-Python full pipeline by default (v4.2.0)
      python -m wikifier health --summary
 3. For sustained agent work: launch `python -m wikifier monitor` (or wikifier.sh monitor) in bg / daemon for 30s "Pruned 0 / No new" heartbeat + auto BRC healing. Use .last_check + health for observability.
 4. Immediately consult the small health matrix (file_health.md or health --summary) + pending_updates.md. Prioritise 🔴 then 🟡. Use this for quick reference instead of full file reads to save tokens.
@@ -103,8 +110,8 @@ Key returns (structured dict primary; "success": bool always present on library
 - `health(project_root=None, directory=None, format="text"|"json"|"summary") -> str | dict`: json includes full entries + "dependency_intel" (acs_summary, etc.) for agent reasoning.
 - `mark_green(file, reason="", project_root=None) -> dict`
 - `suggest_next_actions(..., format="json") -> dict`: { "success", "red", "yellow", "suggestions": list[str], "health_summary", "acs_note" }
-- `update_maps(..., use_python_primary=True, directory=...) -> dict`: delegates to run_full_update result + facade metadata.
-- `run_full_update(...) -> dict`: { "success", "root", "files_to_reparse", "persist_pipeline_exercised", "barrel_creative_tied_in_pure_path", "dirty_sample", ... }
+- `update_maps(..., directory=..., max_files=...) -> dict`: delegates to run_full_update — the real full pipeline (all dirty files parsed in-process, canonical persist, cycles/ACS/reverse deps, atomic library.md) + facade metadata.
+- `run_full_update(...) -> dict`: { "success", "root", "mode", "parseable_files", "files_to_reparse", "files_parsed", "files_skipped", "edges_persisted", "cycles", "library", "dirty_sample", "persist_pipeline_exercised" (compat), ... }
 
 Text formats are for human review only. Agents MUST request json/summary for machine use in loops.
 
@@ -151,7 +158,7 @@ See the full design, mandatory workflow example, and M2 exit criteria in the pla
 | `wikifier prepare-edit` | `<file>` | Stage current mtime before you start editing (for future diffing). |
 | `wikifier mark-green` | `<file> [reason]` | Flip file status to Green after you have written/updated its wiki summary. |
 | `wikifier monitor` | — | Start background 30s heartbeat (run with `&` or in separate terminal). |
-| `wikifier update-maps` | — | Rebuild `library.md` with fresh Mermaid dependency graph + import summary. |
+| `wikifier update-maps` | `[--full] [--directory=...] [--max-files=N] [--sh]` | Rebuild `library.md` + import cache (pure-Python full pipeline by default; `--sh` = legacy shell fallback). |
 | `wikifier validate` | — | Ensure every file in monitored_paths has at least a health row. |
 | `wikifier journal` | `[YYYY-MM-DD]` | Read the journal for a day (default = today). |
 | `wikifier issues` | `[simple|moderate|high|critical]` | List logged issues by severity. |
@@ -164,7 +171,7 @@ This project exposes a first-class MCP server (wikifier-mcp or python -m wikifie
 
 **M5.1+ reality (from dogfood on external 5k-79k+ projects)**: MCP tools are preferred when available (get_project_status, get_file_wiki, get_barrel_reports for deep BRC, check_changes, record_change, mark_green, suggest_next_actions, health equivalents, etc.). **Always pass project_root= (or use WIKIFIER_PROJECT_ROOT env)** for external/user projects (RecipeLab_alt BRC stress, ConsistencyHub, llvm subs, etc.). 
 
-MCP can timeout on large/BRC-heavy targets (M5 alt ~19-20 named yellows from challengeFeatures re-exports, Consistency ~1k, llvm 168k+ units). In that case, **immediately fall back to CLI/library** (python -m wikifier health --summary or the library health(..., format="summary")) — these are reliable and were the workhorse in M5 sustained/monitor work.
+MCP timeouts are rare since v4.2.0 (the library workflow-call deadlock is fixed and barrel persistence is batched), but on very large/BRC-heavy targets they can still happen. In that case, **immediately fall back to CLI/library** (python -m wikifier health --summary or the library health(..., format="summary")) — these are reliable and were the workhorse in M5 sustained/monitor work.
 
 The server implements hardened external discovery (delegates to cli.py discover_project_root / _get_effective_root), 60s timeouts, actionable errors, and parity with library.
 
@@ -215,7 +222,7 @@ record_change("src/api/client.py", "Switched to httpx.AsyncClient because the sy
 mark_green("src/api/client.py", "Purpose + import summary updated to reflect httpx usage and retry logic.")
 
 print(suggest_next_actions(format="json"))
-update_maps(directory="src/", use_python_primary=True)  # if needed
+update_maps(directory="src/")  # if imports changed — full pure-Python pipeline (v4.2.0)
 ```
 
 Fallback (shell or MCP) when library not directly importable in the agent env:
@@ -231,11 +238,11 @@ wikifier record-change "src/api/client.py" "Switched to httpx.AsyncClient becaus
 wikifier mark-green "src/api/client.py" "Purpose + import summary updated to reflect httpx usage and retry logic."
 ```
 
-This protocol (v0.5) + the zero-dependency Python library (core has no deps; MCP is optional via `pip install wikifier[mcp]`) make Wikifier a first-class, low-ambiguity citizen in any LLM-driven development workflow across models and environments. The design ensures the mandatory loop is executable with minimal deviation.
+This protocol (v0.6) + the zero-dependency Python library (core has no deps; MCP is optional via `pip install wikifier[mcp]`) make Wikifier a first-class, low-ambiguity citizen in any LLM-driven development workflow across models and environments. The design ensures the mandatory loop is executable with minimal deviation.
 
 **Explicitly zero-dependency by design** (see pyproject.toml: dependencies = [] ; MCP via optional extra only). All core agent operations (health matrix for token-saving lookup, record-change/mark-green for autonomous wiki updates, check-changes, etc.) work with plain Python + the installed package or source — no external services or heavy deps required.
 
-## Quick Reference — Library Surface (v0.5, M5-updated)
+## Quick Reference — Library Surface (v0.6)
 Core imports (all zero-dep for main paths; support project_root= for external/agent-to-agent use):
 `from wikifier import check_changes, record_change, health, mark_green, suggest_next_actions, update_maps, discover_project_root, run_full_update`
 
@@ -243,4 +250,4 @@ All support `project_root=...` and return structured data (plus side-effecting s
 
 See README.md "Intended Use" for the strict agent-to-agent wiki scope (token saving for lookup + autonomous update/create of wiki entries only). M5 dogfood validated this on real external projects with the exact patterns above. Recent 4.0.1 hygiene (health coerce + superseded prune) further improves reliability of the matrix for agents doing direct lookups/updates.
 
-**Human investigation layer (secondary, opt-in)**: `wikifier init` copies only `index.html` (the clean human wiki viewer) into the target project (the folder where MCP/CLI run). `diagnostics.html` (Wikifier maintainer hub) is no longer copied — it would show the wrong tree (Wikifier's internals) and be stale for the host project. Humans open the local `index.html` (via `python -m http.server` recommended) to see a clean visual of *that project's* agent-maintained wiki: prominent code structure / dependency chart (Mermaid) as the hero, followed by a simple "Files & descriptions" list (paths + short "what this file is about" summaries pulled from the wiki notes), and a lightweight "Browse by folder" tree derived from the health data. It auto-polls for refresh when the monitor or agents update the underlying .md files. Prominent copy buttons export "structure as text" (Mermaid source) and a clean full snapshot (tree + file list + descriptions) — exactly the compact, token-saving view agents use. The default human page is intentionally free of dense agent internals (those live in `diagnostics.html` in the Wikifier source for technical users). If an old diagnostics.html is present, it can be safely deleted. The .md files + MCP/CLI/tools remain the primary SSOT and update mechanism for agents. This layer lets humans (or teams) visually investigate and copy-paste wiki summaries for their own work / LLM chats without touching agent behavior or adding deps.
+**Human investigation layer (secondary, opt-in)**: `wikifier init` copies only `index.html` (the clean human wiki viewer) into the target project (the folder where MCP/CLI run). `diagnostics.html` (Wikifier maintainer hub) is no longer copied — it would show the wrong tree (Wikifier's internals) and be stale for the host project. Humans open the local `index.html` (via `python -m http.server` recommended) to see a clean visual of *that project's* agent-maintained wiki: prominent code structure / dependency chart (Mermaid) as the hero, followed by a simple "Files & descriptions" list (paths + short "what this file is about" summaries pulled from the wiki notes), and a lightweight "Browse by folder" tree derived from the health data. A "Quick actions" toolbar provides one-click copy buttons for the main commands (check-changes, update-maps, monitor &); empty states ("No structure map yet", "No files in the wiki yet") have prominent primary buttons for first-time commands (update-maps prioritized, combined check-changes+update-maps for files). On first open with no map, the `wikifier update-maps` command is auto-copied (sessionStorage-guarded one-time per browser session) and live-wait mode is immediately activated. Buttons use copy + live-wait: they copy the exact command and inject a fixed top "Waiting for `wikifier ...` to produce data... (auto-polling)" banner with aggressive 3s polling that auto-detects when library.md / file_health.* update (from the terminal run) and refreshes the chart/files automatically; includes an "I ran it — refresh now" link in the banner and a success toast on detection. A short explanatory note in the UI clarifies the model: this is a pure static zero-dep viewer (browser JS cannot execute host shell commands due to security sandbox); the auto we provide is copy + immediate live-wait + fast poll so results (trees, files, descriptions) appear automatically after the user pastes/runs in their terminal. "Good enough" acceptance recorded for this copy+live-wait UX. Prominent copy buttons also export "structure as text" (Mermaid source) and a clean full snapshot (tree + file list + descriptions) — exactly the compact, token-saving view agents use. The default human page is intentionally free of dense agent internals (those live in `diagnostics.html` in the Wikifier source for technical users). If an old diagnostics.html is present, it can be safely deleted. The .md files + MCP/CLI/tools remain the primary SSOT and update mechanism for agents. This layer lets humans (or teams) visually investigate and copy-paste wiki summaries for their own work / LLM chats without touching agent behavior or adding deps.