wikimap 0.8.0__tar.gz

wikimap-0.8.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Donghyun Ha
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wikimap-0.8.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: wikimap
+Version: 0.8.0
+Summary: Zero-LLM incremental index and lazy semantic notes for personal knowledge vaults (markdown, HTML, PDF, images)
+Author: Donghyun Ha
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/dhha22/wikimap
+Project-URL: Repository, https://github.com/dhha22/wikimap
+Project-URL: Issues, https://github.com/dhha22/wikimap/issues
+Keywords: knowledge-base,wiki,obsidian,search,index,vault,claude-code
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# wikimap
+
+[![ci](https://github.com/dhha22/wikimap/actions/workflows/ci.yml/badge.svg)](https://github.com/dhha22/wikimap/actions/workflows/ci.yml)
+
+**Zero-LLM incremental index + lazy semantic layer for markdown knowledge bases.**
+
+One Python file. Zero dependencies. Zero LLM cost at build time — always. Sub-second updates, no matter how stale your index is.
+
+Built for AI coding assistants (Claude Code and friends) working against a markdown vault: an Obsidian vault, a team wiki, a folder of specs and plans.
+
+## Why not a knowledge-graph tool or RAG?
+
+Tools like [graphify](https://github.com/Graphify-Labs/graphify) extract entities and relationships with an LLM **at build time** (eager extraction). That buys you inferred connections, but the bill comes on every update: change a doc, pay for re-extraction. Let the index drift for a week and your "incremental" update re-extracts half the corpus. RAG has the same eagerness problem with embeddings, plus a vector store to babysit.
+
+wikimap inverts the design: **eager structure, lazy semantics.**
+
+- **Structure is free.** Titles, headings, wikilinks, markdown links, requirement IDs, code-file references — all extracted by deterministic parsing. No LLM, no embeddings, no API key.
+- **Semantics are earned at answer time.** When your assistant answers a question by synthesizing documents, it saves the conclusion as a *note* pinned to the source files' content hashes. When it confirms an unwritten connection between two docs, that becomes an *edge* pinned to both hashes. Change a source file and the cached knowledge goes stale automatically — it silently disappears from search results instead of feeding the model outdated facts.
+
+The LLM cost is proportional to **what you actually asked**, never to corpus size.
+
+## Measured (262-doc Korean/English vault, M-series Mac, wikimap 0.4.0)
+
+| Operation | wikimap | graphify (same vault, same change set) |
+|---|---|---|
+| Full index build | 0.5 s, $0 | minutes + LLM extraction cost |
+| Update after editing 1 doc + adding 1 + deleting 1 | **0.1 s, 0 tokens** | **~95 s + 46k tokens** (measured), plus community re-labeling |
+| Update after index drifted for days | still sub-second (sha-diff) | re-detected 287 of 306 files as changed → near-full re-extraction |
+| Search recall@5 (10 mixed Korean/English queries) | **10/10**, ~60 ms | 5/10 start-node matches — default query filter drops terms shorter than 4 chars, so every short Korean term is discarded |
+| Search output | section + line number + matched snippet | entity labels; you still re-read the source files |
+| Deleted file cleanup | automatic, verified | 9.7% of source files in the graph were ghosts (already deleted); 40 duplicate node labels |
+| Determinism | same input → byte-identical index | non-deterministic graphs from identical inputs ([upstream #1695](https://github.com/Graphify-Labs/graphify/issues/1695)) |
+
+At scale (same vault duplicated to **3,760 docs**): full build 12 s (one-time — an FTS5 trigram index kicks in at ≥500 docs), incremental update with 3 changes **0.19 s**, search 60–100 ms via FTS5 (vs ~0.3 s linear fallback). Queries containing terms under 3 characters fall back to the exact linear scan, so CJK short-word recall is never sacrificed for speed.
+
+On an expanded 30-query golden set (Korean/English/mixed, 309-doc vault): **recall@5 30/30, avg 67 ms** (re-verified at 30/30 after HTML indexing in 0.5.0, the semantics-file migration in 0.6.0, PDF/image indexing in 0.7.0, and CMap decoding + partial-match fallback in 0.8.0). Ranking changes are gated by this kind of golden set in CI — the test suite (`python3 tests.py`, stdlib only) covers incremental sync, ghost-free deletes, byte-identical determinism, FTS consistency at scale, CJK short-term fallback, ignore config, map relocation, HTML tag-strip indexing, semantics surviving DB deletion, the ≤0.5.x migration path, `--json` schemas, hook append-preservation, phrase/field/tag/type queries, partial-fallback marking, PDF noise exclusion, per-font CMap decoding (CID hex/literal, bfrange, ASCII85+Flate chains, Form XObjects, Type3), image alt indexing, dotted-filename wikilink resolution, `mv` reference rewriting, console-script installs, and install never touching an existing `SKILL.md`. CI runs it on macOS, Linux, and Windows, Python 3.8 and 3.13.
+
+Reproduce on your own vault: `python3 bench.py --root <vault> --cold`, or with your own golden set: `bench.py --root <vault> --queries q.tsv` (lines of `query<TAB>expected-path-substring`).
+
+## Install
+
+```bash
+pipx install wikimap                # or: uv tool install wikimap / pip install wikimap
+cd your-vault && wikimap update
+```
+
+Or copy the single file — same thing, works offline and without pip:
+
+```bash
+curl -O https://raw.githubusercontent.com/dhha22/wikimap/main/wikimap.py
+cd your-vault && python3 wikimap.py update
+```
+
+Either way, `wikimap install` (or `python3 wikimap.py install`) sets it up as a Claude Code skill at `~/.claude/skills/wikimap/`; `install --project` writes to `./.claude` for per-repo setup. Existing `SKILL.md` customizations are never overwritten. Requires Python 3.8+, nothing else.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `update [--ignore <dir\|glob>] [--map-path <rel> \| --no-map]` | Incremental re-index (sha-diff) + regenerate `MAP.md`, the one-page vault map agents read first. Prints coverage — indexed vs skipped counts by extension, so nothing is dropped silently. `MAP.md` ends with a Health section: orphan docs, broken links, stale semantics. Excludes: `.wikimapignore` at the vault root (one dir/glob per line, persistent) or `--ignore` (this run only). `--map-path`/`--no-map` relocate or disable the generated map — persisted in the index |
+| `search "query" [-n 8] [-C 3 \| --full]` | Ranked section search — filename, title, and heading matches boosted; FTS5-accelerated on vaults ≥500 docs. Exact file:line + matched lines (≤3). `-C N` adds N context lines, `--full` prints the whole section. Fresh notes surface first. Query syntax: `"exact phrase"`, `title:` / `path:` / `heading:` / `tag:` field filters (frontmatter `tags: [a, b]` are indexed and summarized in the map), and `type:md\|html\|pdf\|image\|text` file-type filter. When no section matches every term, results relax to a majority-of-terms OR marked `partial k/n` — never mixed with full matches; field filters stay hard |
+| `links <target>` | Outlinks, backlinks, and inferred connections of a doc; or every doc mentioning a `REQ-nn` ID. Trust tags on every entry: `[linked\|…]` = a human wrote it in the source, `[inferred\|…]` = guessed then confirmed, sha-verified |
+| `path <a> <b>` | Shortest connection path between two docs — BFS over wiki/markdown links (both directions) plus fresh inferred edges |
+| `note add` | Save an answer-time insight, pinned to source content hashes |
+| `suggest [--doc path] [--wikilink]` | Heuristic candidates for unwritten connections: shared rare terms, shared requirement IDs, shared code references. 0.2 s, no LLM. `--wikilink` prints paste-ready `[[links]]` — promote real connections into the doc body, where every tool can read them |
+| `edge add` | Confirm a connection (agent judges `suggest` candidates); pinned to both files' hashes |
+| `edge repin --src a --dst b` | An edge went stale because an endpoint was edited, but the connection still holds? Refresh the sha pins and keep the rationale — no retyping |
+| `notes` / `edges` `[--all] [--prune]` | List cached semantics; stale entries are hidden by default and prunable |
+| `import-graphify <graph.json>` | One-time migration of INFERRED edges from an existing graphify graph — with hash freshness retrofitted |
+| `install --hook` | Git post-commit hook that runs `update` after every commit — appends to an existing hook, never replaces it |
+| `mv <old> <new> [--apply]` | Move/rename a doc and rewrite every wikilink, markdown, and image reference to it — including the moved file's own relative links and `semantics.jsonl` paths (content hash unchanged, so pinned semantics stay fresh). Dry run unless `--apply` |
+| `fix-links [--json]` | For each broken link the Health section counts: suggest close-match targets. Suggestions only — nothing is auto-applied |
+
+`search`, `links`, `path`, `suggest`, `notes`, and `edges` all take **`--json`** — structured output for agents and scripts, no regex-scraping of human output. Schemas are stable and covered by the test suite.
+
+## How inferred connections work without eager LLM extraction
+
+1. `suggest` proposes candidate pairs from free signals: rare terms shared by only 2–4 documents, shared requirement IDs, references to the same source files. Pairs already linked explicitly are excluded; cross-directory pairs get a boost (more surprising).
+2. Your assistant reads only the top candidates for the doc that changed and confirms the real ones with `edge add`. Cost scales with the edit, not the corpus.
+3. Confirmed edges appear in `links` output and `MAP.md`, and go stale automatically when either endpoint changes. Stale-because-edited but still valid? `edge repin` re-pins it after review, rationale intact.
+
+## Outputs
+
+- `MAP.md` — vault root. Directory taxonomy, hub documents, recent changes, cross-document requirement IDs, inferred connections, fresh notes. The agent entry point.
+- `.wikimap/semantics.jsonl` — the notes and edges themselves, append-only JSON lines. **This file is the source of truth** for the semantic layer: commit it to git to back up and share what your assistant has learned about the vault. Hand-editable; one bad line never takes the layer down.
+- `.wikimap/index.db` — SQLite. A derived cache, genuinely disposable: delete it anytime, `update` rebuilds it from your files plus `semantics.jsonl` with nothing lost.
+
+Upgrading from ≤0.5.x: the first run migrates existing DB notes/edges into `semantics.jsonl` automatically, one-time, nothing to do.
+
+## Coexisting with other vault tools
+
+wikimap is a standalone library — it assumes nothing about what else manages your folder. If another app (Obsidian, a second-brain app with its own index, a static-site generator) also watches the same root, three knobs keep the two from stepping on each other:
+
+- **`.wikimapignore`** — one dir name or glob per line at the vault root. Keeps the other tool's artifacts (trash folders, build output) out of wikimap's index. `.trash/`, `.obsidian/`, and common build dirs are already excluded by default.
+- **`--map-path .wikimap/MAP.md`** — if the other tool indexes markdown at the root, a generated `MAP.md` there would pollute its graph as a giant hub node. Relocating it into `.wikimap/` (which the other tool should skip anyway) hides it from everyone but your agent. Or `--no-map` to skip generation entirely. Both persist across runs.
+- **`suggest --wikilink`** — when confirming discovered connections, prefer pasting explicit `[[links]]` into the document body over `edge add`. Files are the source of truth; explicit links are the one connection format every vault tool understands.
+
+## Scope
+
+wikimap's goal is that **every document in the folder is findable — whatever its format** — plus a relationship layer on top. Currently indexed:
+
+- **Markdown** — the core: frontmatter (`title`, `tags`), headings, wikilinks, md links.
+- **Plain-text prose** (`.txt`, `.rst`, `.org`, `.adoc`) — sectioned by paragraph blocks.
+- **HTML** (`.html`, `.htm`) — tag-stripped, `<title>`/`<h1>` as title, sectioned by heading tags; `<a href>` anchors to local docs join the link graph, `<script>`/`<style>` excluded.
+- **PDF** — deterministic text extraction with stdlib only. Per-font **ToUnicode CMap decoding** handles CID-encoded (most CJK) and subset-font PDFs: the Page→Resources→Font object chain is resolved per font (never unioned — subset code spaces collide), Form XObjects are traversed, 1- and 2-byte code spaces and `[/ASCII85Decode /FlateDecode]` filter chains are supported, and each content stream becomes a search section (a slide/page is the natural unit). PDFs that still don't decode (scanned images) fall back to raw literal-string harvest, then **name+path indexing** — and the update line says so explicitly: no OCR, no silent garbage; every rung is noise-gated.
+- **Images** (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`) — no content analysis; indexed by filename plus every **alt text** that references them (`![alt](img.png)`, `<img alt=…>`), and image references join the link graph. "Where is that checkout-flow diagram?" resolves by name or alt. `.svg` additionally contributes its `<title>`/`<desc>`/text nodes.
+
+It does not parse code ASTs — if you need a call graph of a codebase, use a code-aware tool. It shines where your corpus is prose with structure: specs, policies, plans, notes, research.
+
+## License
+
+MIT

wikimap-0.8.0/README.md

@@ -0,0 +1,112 @@
+# wikimap
+
+[![ci](https://github.com/dhha22/wikimap/actions/workflows/ci.yml/badge.svg)](https://github.com/dhha22/wikimap/actions/workflows/ci.yml)
+
+**Zero-LLM incremental index + lazy semantic layer for markdown knowledge bases.**
+
+One Python file. Zero dependencies. Zero LLM cost at build time — always. Sub-second updates, no matter how stale your index is.
+
+Built for AI coding assistants (Claude Code and friends) working against a markdown vault: an Obsidian vault, a team wiki, a folder of specs and plans.
+
+## Why not a knowledge-graph tool or RAG?
+
+Tools like [graphify](https://github.com/Graphify-Labs/graphify) extract entities and relationships with an LLM **at build time** (eager extraction). That buys you inferred connections, but the bill comes on every update: change a doc, pay for re-extraction. Let the index drift for a week and your "incremental" update re-extracts half the corpus. RAG has the same eagerness problem with embeddings, plus a vector store to babysit.
+
+wikimap inverts the design: **eager structure, lazy semantics.**
+
+- **Structure is free.** Titles, headings, wikilinks, markdown links, requirement IDs, code-file references — all extracted by deterministic parsing. No LLM, no embeddings, no API key.
+- **Semantics are earned at answer time.** When your assistant answers a question by synthesizing documents, it saves the conclusion as a *note* pinned to the source files' content hashes. When it confirms an unwritten connection between two docs, that becomes an *edge* pinned to both hashes. Change a source file and the cached knowledge goes stale automatically — it silently disappears from search results instead of feeding the model outdated facts.
+
+The LLM cost is proportional to **what you actually asked**, never to corpus size.
+
+## Measured (262-doc Korean/English vault, M-series Mac, wikimap 0.4.0)
+
+| Operation | wikimap | graphify (same vault, same change set) |
+|---|---|---|
+| Full index build | 0.5 s, $0 | minutes + LLM extraction cost |
+| Update after editing 1 doc + adding 1 + deleting 1 | **0.1 s, 0 tokens** | **~95 s + 46k tokens** (measured), plus community re-labeling |
+| Update after index drifted for days | still sub-second (sha-diff) | re-detected 287 of 306 files as changed → near-full re-extraction |
+| Search recall@5 (10 mixed Korean/English queries) | **10/10**, ~60 ms | 5/10 start-node matches — default query filter drops terms shorter than 4 chars, so every short Korean term is discarded |
+| Search output | section + line number + matched snippet | entity labels; you still re-read the source files |
+| Deleted file cleanup | automatic, verified | 9.7% of source files in the graph were ghosts (already deleted); 40 duplicate node labels |
+| Determinism | same input → byte-identical index | non-deterministic graphs from identical inputs ([upstream #1695](https://github.com/Graphify-Labs/graphify/issues/1695)) |
+
+At scale (same vault duplicated to **3,760 docs**): full build 12 s (one-time — an FTS5 trigram index kicks in at ≥500 docs), incremental update with 3 changes **0.19 s**, search 60–100 ms via FTS5 (vs ~0.3 s linear fallback). Queries containing terms under 3 characters fall back to the exact linear scan, so CJK short-word recall is never sacrificed for speed.
+
+On an expanded 30-query golden set (Korean/English/mixed, 309-doc vault): **recall@5 30/30, avg 67 ms** (re-verified at 30/30 after HTML indexing in 0.5.0, the semantics-file migration in 0.6.0, PDF/image indexing in 0.7.0, and CMap decoding + partial-match fallback in 0.8.0). Ranking changes are gated by this kind of golden set in CI — the test suite (`python3 tests.py`, stdlib only) covers incremental sync, ghost-free deletes, byte-identical determinism, FTS consistency at scale, CJK short-term fallback, ignore config, map relocation, HTML tag-strip indexing, semantics surviving DB deletion, the ≤0.5.x migration path, `--json` schemas, hook append-preservation, phrase/field/tag/type queries, partial-fallback marking, PDF noise exclusion, per-font CMap decoding (CID hex/literal, bfrange, ASCII85+Flate chains, Form XObjects, Type3), image alt indexing, dotted-filename wikilink resolution, `mv` reference rewriting, console-script installs, and install never touching an existing `SKILL.md`. CI runs it on macOS, Linux, and Windows, Python 3.8 and 3.13.
+
+Reproduce on your own vault: `python3 bench.py --root <vault> --cold`, or with your own golden set: `bench.py --root <vault> --queries q.tsv` (lines of `query<TAB>expected-path-substring`).
+
+## Install
+
+```bash
+pipx install wikimap                # or: uv tool install wikimap / pip install wikimap
+cd your-vault && wikimap update
+```
+
+Or copy the single file — same thing, works offline and without pip:
+
+```bash
+curl -O https://raw.githubusercontent.com/dhha22/wikimap/main/wikimap.py
+cd your-vault && python3 wikimap.py update
+```
+
+Either way, `wikimap install` (or `python3 wikimap.py install`) sets it up as a Claude Code skill at `~/.claude/skills/wikimap/`; `install --project` writes to `./.claude` for per-repo setup. Existing `SKILL.md` customizations are never overwritten. Requires Python 3.8+, nothing else.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `update [--ignore <dir\|glob>] [--map-path <rel> \| --no-map]` | Incremental re-index (sha-diff) + regenerate `MAP.md`, the one-page vault map agents read first. Prints coverage — indexed vs skipped counts by extension, so nothing is dropped silently. `MAP.md` ends with a Health section: orphan docs, broken links, stale semantics. Excludes: `.wikimapignore` at the vault root (one dir/glob per line, persistent) or `--ignore` (this run only). `--map-path`/`--no-map` relocate or disable the generated map — persisted in the index |
+| `search "query" [-n 8] [-C 3 \| --full]` | Ranked section search — filename, title, and heading matches boosted; FTS5-accelerated on vaults ≥500 docs. Exact file:line + matched lines (≤3). `-C N` adds N context lines, `--full` prints the whole section. Fresh notes surface first. Query syntax: `"exact phrase"`, `title:` / `path:` / `heading:` / `tag:` field filters (frontmatter `tags: [a, b]` are indexed and summarized in the map), and `type:md\|html\|pdf\|image\|text` file-type filter. When no section matches every term, results relax to a majority-of-terms OR marked `partial k/n` — never mixed with full matches; field filters stay hard |
+| `links <target>` | Outlinks, backlinks, and inferred connections of a doc; or every doc mentioning a `REQ-nn` ID. Trust tags on every entry: `[linked\|…]` = a human wrote it in the source, `[inferred\|…]` = guessed then confirmed, sha-verified |
+| `path <a> <b>` | Shortest connection path between two docs — BFS over wiki/markdown links (both directions) plus fresh inferred edges |
+| `note add` | Save an answer-time insight, pinned to source content hashes |
+| `suggest [--doc path] [--wikilink]` | Heuristic candidates for unwritten connections: shared rare terms, shared requirement IDs, shared code references. 0.2 s, no LLM. `--wikilink` prints paste-ready `[[links]]` — promote real connections into the doc body, where every tool can read them |
+| `edge add` | Confirm a connection (agent judges `suggest` candidates); pinned to both files' hashes |
+| `edge repin --src a --dst b` | An edge went stale because an endpoint was edited, but the connection still holds? Refresh the sha pins and keep the rationale — no retyping |
+| `notes` / `edges` `[--all] [--prune]` | List cached semantics; stale entries are hidden by default and prunable |
+| `import-graphify <graph.json>` | One-time migration of INFERRED edges from an existing graphify graph — with hash freshness retrofitted |
+| `install --hook` | Git post-commit hook that runs `update` after every commit — appends to an existing hook, never replaces it |
+| `mv <old> <new> [--apply]` | Move/rename a doc and rewrite every wikilink, markdown, and image reference to it — including the moved file's own relative links and `semantics.jsonl` paths (content hash unchanged, so pinned semantics stay fresh). Dry run unless `--apply` |
+| `fix-links [--json]` | For each broken link the Health section counts: suggest close-match targets. Suggestions only — nothing is auto-applied |
+
+`search`, `links`, `path`, `suggest`, `notes`, and `edges` all take **`--json`** — structured output for agents and scripts, no regex-scraping of human output. Schemas are stable and covered by the test suite.
+
+## How inferred connections work without eager LLM extraction
+
+1. `suggest` proposes candidate pairs from free signals: rare terms shared by only 2–4 documents, shared requirement IDs, references to the same source files. Pairs already linked explicitly are excluded; cross-directory pairs get a boost (more surprising).
+2. Your assistant reads only the top candidates for the doc that changed and confirms the real ones with `edge add`. Cost scales with the edit, not the corpus.
+3. Confirmed edges appear in `links` output and `MAP.md`, and go stale automatically when either endpoint changes. Stale-because-edited but still valid? `edge repin` re-pins it after review, rationale intact.
+
+## Outputs
+
+- `MAP.md` — vault root. Directory taxonomy, hub documents, recent changes, cross-document requirement IDs, inferred connections, fresh notes. The agent entry point.
+- `.wikimap/semantics.jsonl` — the notes and edges themselves, append-only JSON lines. **This file is the source of truth** for the semantic layer: commit it to git to back up and share what your assistant has learned about the vault. Hand-editable; one bad line never takes the layer down.
+- `.wikimap/index.db` — SQLite. A derived cache, genuinely disposable: delete it anytime, `update` rebuilds it from your files plus `semantics.jsonl` with nothing lost.
+
+Upgrading from ≤0.5.x: the first run migrates existing DB notes/edges into `semantics.jsonl` automatically, one-time, nothing to do.
+
+## Coexisting with other vault tools
+
+wikimap is a standalone library — it assumes nothing about what else manages your folder. If another app (Obsidian, a second-brain app with its own index, a static-site generator) also watches the same root, three knobs keep the two from stepping on each other:
+
+- **`.wikimapignore`** — one dir name or glob per line at the vault root. Keeps the other tool's artifacts (trash folders, build output) out of wikimap's index. `.trash/`, `.obsidian/`, and common build dirs are already excluded by default.
+- **`--map-path .wikimap/MAP.md`** — if the other tool indexes markdown at the root, a generated `MAP.md` there would pollute its graph as a giant hub node. Relocating it into `.wikimap/` (which the other tool should skip anyway) hides it from everyone but your agent. Or `--no-map` to skip generation entirely. Both persist across runs.
+- **`suggest --wikilink`** — when confirming discovered connections, prefer pasting explicit `[[links]]` into the document body over `edge add`. Files are the source of truth; explicit links are the one connection format every vault tool understands.
+
+## Scope
+
+wikimap's goal is that **every document in the folder is findable — whatever its format** — plus a relationship layer on top. Currently indexed:
+
+- **Markdown** — the core: frontmatter (`title`, `tags`), headings, wikilinks, md links.
+- **Plain-text prose** (`.txt`, `.rst`, `.org`, `.adoc`) — sectioned by paragraph blocks.
+- **HTML** (`.html`, `.htm`) — tag-stripped, `<title>`/`<h1>` as title, sectioned by heading tags; `<a href>` anchors to local docs join the link graph, `<script>`/`<style>` excluded.
+- **PDF** — deterministic text extraction with stdlib only. Per-font **ToUnicode CMap decoding** handles CID-encoded (most CJK) and subset-font PDFs: the Page→Resources→Font object chain is resolved per font (never unioned — subset code spaces collide), Form XObjects are traversed, 1- and 2-byte code spaces and `[/ASCII85Decode /FlateDecode]` filter chains are supported, and each content stream becomes a search section (a slide/page is the natural unit). PDFs that still don't decode (scanned images) fall back to raw literal-string harvest, then **name+path indexing** — and the update line says so explicitly: no OCR, no silent garbage; every rung is noise-gated.
+- **Images** (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`) — no content analysis; indexed by filename plus every **alt text** that references them (`![alt](img.png)`, `<img alt=…>`), and image references join the link graph. "Where is that checkout-flow diagram?" resolves by name or alt. `.svg` additionally contributes its `<title>`/`<desc>`/text nodes.
+
+It does not parse code ASTs — if you need a call graph of a codebase, use a code-aware tool. It shines where your corpus is prose with structure: specs, policies, plans, notes, research.
+
+## License
+
+MIT

wikimap-0.8.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "wikimap"
+dynamic = ["version"]
+description = "Zero-LLM incremental index and lazy semantic notes for personal knowledge vaults (markdown, HTML, PDF, images)"
+readme = "README.md"
+requires-python = ">=3.8"
+license = "MIT"
+authors = [{ name = "Donghyun Ha" }]
+keywords = ["knowledge-base", "wiki", "obsidian", "search", "index", "vault", "claude-code"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: Text Processing :: Indexing",
+]
+
+[project.scripts]
+wikimap = "wikimap:main"
+
+[project.urls]
+Homepage = "https://github.com/dhha22/wikimap"
+Repository = "https://github.com/dhha22/wikimap"
+Issues = "https://github.com/dhha22/wikimap/issues"
+
+[tool.setuptools]
+py-modules = ["wikimap"]
+
+[tool.setuptools.dynamic]
+version = { attr = "wikimap.VERSION" }

wikimap-0.8.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

wikimap-0.8.0/wikimap.egg-info/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: wikimap
+Version: 0.8.0
+Summary: Zero-LLM incremental index and lazy semantic notes for personal knowledge vaults (markdown, HTML, PDF, images)
+Author: Donghyun Ha
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/dhha22/wikimap
+Project-URL: Repository, https://github.com/dhha22/wikimap
+Project-URL: Issues, https://github.com/dhha22/wikimap/issues
+Keywords: knowledge-base,wiki,obsidian,search,index,vault,claude-code
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# wikimap
+
+[![ci](https://github.com/dhha22/wikimap/actions/workflows/ci.yml/badge.svg)](https://github.com/dhha22/wikimap/actions/workflows/ci.yml)
+
+**Zero-LLM incremental index + lazy semantic layer for markdown knowledge bases.**
+
+One Python file. Zero dependencies. Zero LLM cost at build time — always. Sub-second updates, no matter how stale your index is.
+
+Built for AI coding assistants (Claude Code and friends) working against a markdown vault: an Obsidian vault, a team wiki, a folder of specs and plans.
+
+## Why not a knowledge-graph tool or RAG?
+
+Tools like [graphify](https://github.com/Graphify-Labs/graphify) extract entities and relationships with an LLM **at build time** (eager extraction). That buys you inferred connections, but the bill comes on every update: change a doc, pay for re-extraction. Let the index drift for a week and your "incremental" update re-extracts half the corpus. RAG has the same eagerness problem with embeddings, plus a vector store to babysit.
+
+wikimap inverts the design: **eager structure, lazy semantics.**
+
+- **Structure is free.** Titles, headings, wikilinks, markdown links, requirement IDs, code-file references — all extracted by deterministic parsing. No LLM, no embeddings, no API key.
+- **Semantics are earned at answer time.** When your assistant answers a question by synthesizing documents, it saves the conclusion as a *note* pinned to the source files' content hashes. When it confirms an unwritten connection between two docs, that becomes an *edge* pinned to both hashes. Change a source file and the cached knowledge goes stale automatically — it silently disappears from search results instead of feeding the model outdated facts.
+
+The LLM cost is proportional to **what you actually asked**, never to corpus size.
+
+## Measured (262-doc Korean/English vault, M-series Mac, wikimap 0.4.0)
+
+| Operation | wikimap | graphify (same vault, same change set) |
+|---|---|---|
+| Full index build | 0.5 s, $0 | minutes + LLM extraction cost |
+| Update after editing 1 doc + adding 1 + deleting 1 | **0.1 s, 0 tokens** | **~95 s + 46k tokens** (measured), plus community re-labeling |
+| Update after index drifted for days | still sub-second (sha-diff) | re-detected 287 of 306 files as changed → near-full re-extraction |
+| Search recall@5 (10 mixed Korean/English queries) | **10/10**, ~60 ms | 5/10 start-node matches — default query filter drops terms shorter than 4 chars, so every short Korean term is discarded |
+| Search output | section + line number + matched snippet | entity labels; you still re-read the source files |
+| Deleted file cleanup | automatic, verified | 9.7% of source files in the graph were ghosts (already deleted); 40 duplicate node labels |
+| Determinism | same input → byte-identical index | non-deterministic graphs from identical inputs ([upstream #1695](https://github.com/Graphify-Labs/graphify/issues/1695)) |
+
+At scale (same vault duplicated to **3,760 docs**): full build 12 s (one-time — an FTS5 trigram index kicks in at ≥500 docs), incremental update with 3 changes **0.19 s**, search 60–100 ms via FTS5 (vs ~0.3 s linear fallback). Queries containing terms under 3 characters fall back to the exact linear scan, so CJK short-word recall is never sacrificed for speed.
+
+On an expanded 30-query golden set (Korean/English/mixed, 309-doc vault): **recall@5 30/30, avg 67 ms** (re-verified at 30/30 after HTML indexing in 0.5.0, the semantics-file migration in 0.6.0, PDF/image indexing in 0.7.0, and CMap decoding + partial-match fallback in 0.8.0). Ranking changes are gated by this kind of golden set in CI — the test suite (`python3 tests.py`, stdlib only) covers incremental sync, ghost-free deletes, byte-identical determinism, FTS consistency at scale, CJK short-term fallback, ignore config, map relocation, HTML tag-strip indexing, semantics surviving DB deletion, the ≤0.5.x migration path, `--json` schemas, hook append-preservation, phrase/field/tag/type queries, partial-fallback marking, PDF noise exclusion, per-font CMap decoding (CID hex/literal, bfrange, ASCII85+Flate chains, Form XObjects, Type3), image alt indexing, dotted-filename wikilink resolution, `mv` reference rewriting, console-script installs, and install never touching an existing `SKILL.md`. CI runs it on macOS, Linux, and Windows, Python 3.8 and 3.13.
+
+Reproduce on your own vault: `python3 bench.py --root <vault> --cold`, or with your own golden set: `bench.py --root <vault> --queries q.tsv` (lines of `query<TAB>expected-path-substring`).
+
+## Install
+
+```bash
+pipx install wikimap                # or: uv tool install wikimap / pip install wikimap
+cd your-vault && wikimap update
+```
+
+Or copy the single file — same thing, works offline and without pip:
+
+```bash
+curl -O https://raw.githubusercontent.com/dhha22/wikimap/main/wikimap.py
+cd your-vault && python3 wikimap.py update
+```
+
+Either way, `wikimap install` (or `python3 wikimap.py install`) sets it up as a Claude Code skill at `~/.claude/skills/wikimap/`; `install --project` writes to `./.claude` for per-repo setup. Existing `SKILL.md` customizations are never overwritten. Requires Python 3.8+, nothing else.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `update [--ignore <dir\|glob>] [--map-path <rel> \| --no-map]` | Incremental re-index (sha-diff) + regenerate `MAP.md`, the one-page vault map agents read first. Prints coverage — indexed vs skipped counts by extension, so nothing is dropped silently. `MAP.md` ends with a Health section: orphan docs, broken links, stale semantics. Excludes: `.wikimapignore` at the vault root (one dir/glob per line, persistent) or `--ignore` (this run only). `--map-path`/`--no-map` relocate or disable the generated map — persisted in the index |
+| `search "query" [-n 8] [-C 3 \| --full]` | Ranked section search — filename, title, and heading matches boosted; FTS5-accelerated on vaults ≥500 docs. Exact file:line + matched lines (≤3). `-C N` adds N context lines, `--full` prints the whole section. Fresh notes surface first. Query syntax: `"exact phrase"`, `title:` / `path:` / `heading:` / `tag:` field filters (frontmatter `tags: [a, b]` are indexed and summarized in the map), and `type:md\|html\|pdf\|image\|text` file-type filter. When no section matches every term, results relax to a majority-of-terms OR marked `partial k/n` — never mixed with full matches; field filters stay hard |
+| `links <target>` | Outlinks, backlinks, and inferred connections of a doc; or every doc mentioning a `REQ-nn` ID. Trust tags on every entry: `[linked\|…]` = a human wrote it in the source, `[inferred\|…]` = guessed then confirmed, sha-verified |
+| `path <a> <b>` | Shortest connection path between two docs — BFS over wiki/markdown links (both directions) plus fresh inferred edges |
+| `note add` | Save an answer-time insight, pinned to source content hashes |
+| `suggest [--doc path] [--wikilink]` | Heuristic candidates for unwritten connections: shared rare terms, shared requirement IDs, shared code references. 0.2 s, no LLM. `--wikilink` prints paste-ready `[[links]]` — promote real connections into the doc body, where every tool can read them |
+| `edge add` | Confirm a connection (agent judges `suggest` candidates); pinned to both files' hashes |
+| `edge repin --src a --dst b` | An edge went stale because an endpoint was edited, but the connection still holds? Refresh the sha pins and keep the rationale — no retyping |
+| `notes` / `edges` `[--all] [--prune]` | List cached semantics; stale entries are hidden by default and prunable |
+| `import-graphify <graph.json>` | One-time migration of INFERRED edges from an existing graphify graph — with hash freshness retrofitted |
+| `install --hook` | Git post-commit hook that runs `update` after every commit — appends to an existing hook, never replaces it |
+| `mv <old> <new> [--apply]` | Move/rename a doc and rewrite every wikilink, markdown, and image reference to it — including the moved file's own relative links and `semantics.jsonl` paths (content hash unchanged, so pinned semantics stay fresh). Dry run unless `--apply` |
+| `fix-links [--json]` | For each broken link the Health section counts: suggest close-match targets. Suggestions only — nothing is auto-applied |
+
+`search`, `links`, `path`, `suggest`, `notes`, and `edges` all take **`--json`** — structured output for agents and scripts, no regex-scraping of human output. Schemas are stable and covered by the test suite.
+
+## How inferred connections work without eager LLM extraction
+
+1. `suggest` proposes candidate pairs from free signals: rare terms shared by only 2–4 documents, shared requirement IDs, references to the same source files. Pairs already linked explicitly are excluded; cross-directory pairs get a boost (more surprising).
+2. Your assistant reads only the top candidates for the doc that changed and confirms the real ones with `edge add`. Cost scales with the edit, not the corpus.
+3. Confirmed edges appear in `links` output and `MAP.md`, and go stale automatically when either endpoint changes. Stale-because-edited but still valid? `edge repin` re-pins it after review, rationale intact.
+
+## Outputs
+
+- `MAP.md` — vault root. Directory taxonomy, hub documents, recent changes, cross-document requirement IDs, inferred connections, fresh notes. The agent entry point.
+- `.wikimap/semantics.jsonl` — the notes and edges themselves, append-only JSON lines. **This file is the source of truth** for the semantic layer: commit it to git to back up and share what your assistant has learned about the vault. Hand-editable; one bad line never takes the layer down.
+- `.wikimap/index.db` — SQLite. A derived cache, genuinely disposable: delete it anytime, `update` rebuilds it from your files plus `semantics.jsonl` with nothing lost.
+
+Upgrading from ≤0.5.x: the first run migrates existing DB notes/edges into `semantics.jsonl` automatically, one-time, nothing to do.
+
+## Coexisting with other vault tools
+
+wikimap is a standalone library — it assumes nothing about what else manages your folder. If another app (Obsidian, a second-brain app with its own index, a static-site generator) also watches the same root, three knobs keep the two from stepping on each other:
+
+- **`.wikimapignore`** — one dir name or glob per line at the vault root. Keeps the other tool's artifacts (trash folders, build output) out of wikimap's index. `.trash/`, `.obsidian/`, and common build dirs are already excluded by default.
+- **`--map-path .wikimap/MAP.md`** — if the other tool indexes markdown at the root, a generated `MAP.md` there would pollute its graph as a giant hub node. Relocating it into `.wikimap/` (which the other tool should skip anyway) hides it from everyone but your agent. Or `--no-map` to skip generation entirely. Both persist across runs.
+- **`suggest --wikilink`** — when confirming discovered connections, prefer pasting explicit `[[links]]` into the document body over `edge add`. Files are the source of truth; explicit links are the one connection format every vault tool understands.
+
+## Scope
+
+wikimap's goal is that **every document in the folder is findable — whatever its format** — plus a relationship layer on top. Currently indexed:
+
+- **Markdown** — the core: frontmatter (`title`, `tags`), headings, wikilinks, md links.
+- **Plain-text prose** (`.txt`, `.rst`, `.org`, `.adoc`) — sectioned by paragraph blocks.
+- **HTML** (`.html`, `.htm`) — tag-stripped, `<title>`/`<h1>` as title, sectioned by heading tags; `<a href>` anchors to local docs join the link graph, `<script>`/`<style>` excluded.
+- **PDF** — deterministic text extraction with stdlib only. Per-font **ToUnicode CMap decoding** handles CID-encoded (most CJK) and subset-font PDFs: the Page→Resources→Font object chain is resolved per font (never unioned — subset code spaces collide), Form XObjects are traversed, 1- and 2-byte code spaces and `[/ASCII85Decode /FlateDecode]` filter chains are supported, and each content stream becomes a search section (a slide/page is the natural unit). PDFs that still don't decode (scanned images) fall back to raw literal-string harvest, then **name+path indexing** — and the update line says so explicitly: no OCR, no silent garbage; every rung is noise-gated.
+- **Images** (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`) — no content analysis; indexed by filename plus every **alt text** that references them (`![alt](img.png)`, `<img alt=…>`), and image references join the link graph. "Where is that checkout-flow diagram?" resolves by name or alt. `.svg` additionally contributes its `<title>`/`<desc>`/text nodes.
+
+It does not parse code ASTs — if you need a call graph of a codebase, use a code-aware tool. It shines where your corpus is prose with structure: specs, policies, plans, notes, research.
+
+## License
+
+MIT

wikimap-0.8.0/wikimap.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+LICENSE
+README.md
+pyproject.toml
+wikimap.py
+wikimap.egg-info/PKG-INFO
+wikimap.egg-info/SOURCES.txt
+wikimap.egg-info/dependency_links.txt
+wikimap.egg-info/entry_points.txt
+wikimap.egg-info/top_level.txt

wikimap-0.8.0/wikimap.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

wikimap-0.8.0/wikimap.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+wikimap = wikimap:main

wikimap-0.8.0/wikimap.egg-info/top_level.txt

@@ -0,0 +1 @@
+wikimap