graphmark 0.8.0__tar.gz → 0.8.1__tar.gz

{graphmark-0.8.0 → graphmark-0.8.1}/CHANGELOG.md

@@ -1,6 +1,34 @@
 # CHANGELOG
 
 
+## v0.8.1 (2026-07-26)
+
+### Performance Improvements
+
+- **graph**: Index path resolution by filename instead of scanning (#157)
+  ([#160](https://github.com/cdcoonce/graphmark/pull/160),
+  [`84cdd29`](https://github.com/cdcoonce/graphmark/commit/84cdd297a665d93ffa3cbb988c18550ea57f5c94))
+
+Every path-qualified display flattened the catalog and compared against every rel_path. That ceiling
+  was documented and accepted while the branch was rare — most wikilink displays are bare names
+  hitting a dict — but #152 made it the only branch, since every markdown link is path-qualified by
+  construction. blue-book (1120 notes, 10,149 markdown links) took 2.7 s against 0.25 s for the same
+  vault read as wikilinks.
+
+A legal suffix match must consume whole path components, so a candidate's last component always
+  equals the display's last plus ".md". Bucketing by that filename is exactly equivalent to scanning
+  — same matches, same order, same verdicts.
+
+The first attempt was no faster, and profiling rather than guessing found why: `_diagnose` consults
+  the in-scope catalog AND the out-of-scope one for the same display, so a single-slot cache
+  alternated and missed every time — 29,265 index rebuilds for 10,149 links. The cache is now
+  bounded-multi-slot, and it holds each catalog rather than just its id(), so a freed dict cannot
+  have its address reused under a stale index.
+
+blue-book markdown: 2718 ms -> 267 ms (10.2x), now level with wikilink mode. Reference vault 195 ->
+  181 ms. All link counts byte-identical in every mode.
+
+
 ## v0.8.0 (2026-07-26)
 
 ### Documentation

{graphmark-0.8.0 → graphmark-0.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graphmark
-Version: 0.8.0
+Version: 0.8.1
 Summary: Deterministic knowledge-graph analysis for markdown / [[wikilink]] vaults.
 Project-URL: Homepage, https://github.com/cdcoonce/graphmark
 Project-URL: Repository, https://github.com/cdcoonce/graphmark

{graphmark-0.8.0 → graphmark-0.8.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "graphmark"
-version = "0.8.0"
+version = "0.8.1"
 description = "Deterministic knowledge-graph analysis for markdown / [[wikilink]] vaults."
 readme = "README.md"
 requires-python = ">=3.11"

{graphmark-0.8.0 → graphmark-0.8.1}/src/graphmark/graph.py

@@ -144,12 +144,7 @@ def candidates_for(display: str, catalog: dict[str, list[str]]) -> list[str]:
         return []
     if "/" in target:
         suffix = _fold_case(target) + ".md"
-        return sorted(
-            path
-            for paths in catalog.values()
-            for path in paths
-            if _matches_path_suffix(path, suffix)
-        )
+        return sorted(_suffix_matches(suffix, catalog, _CANDIDATE_INDEX))
     return list(catalog.get(_normalize(target), ()))
 
 
@@ -217,25 +212,74 @@ def build_catalog(docs: list[Document]) -> dict[str, list[str]]:
     return catalog
 
 
+class _FilenameIndex:
+    """Catalog paths bucketed by folded filename, so a path-qualified link scans a bucket.
+
+    A legal suffix match must consume whole path components (see ``_matches_path_suffix``), so a
+    candidate's **last** component always equals the display's last component plus ``.md``.
+    Bucketing by that filename is therefore *exactly* equivalent to scanning every path — same
+    matches, same order — while turning an O(N) sweep into a dict lookup over a bucket that is
+    almost always one entry deep.
+
+    This mattered little while path-qualified links were rare: most wikilink displays are bare names
+    hitting the catalog dict directly, which is why the ``O(L·N)`` ceiling was documented and
+    accepted. #152 changed which branch is hot — every markdown link is path-qualified by
+    construction — and a 1120-note vault with 10,149 of them took 2.9 s against 0.3 s for the same
+    vault read as wikilinks.
+
+    Keyed by catalog identity, which is safe because a catalog is invariant for a whole
+    ``VaultGraph.build()``. Each catalog is **held**, not just its ``id()``: without a reference the
+    dict could be freed and a new one allocated at the same address, and the stale index would then
+    answer for it.
+
+    Several slots, not one, and that is load-bearing rather than defensive. ``_diagnose`` consults
+    two different mappings for the same display — the in-scope ``catalog`` and the ``out_of_scope``
+    one — so a single slot alternates between them and misses *every* time. Measured: a single-slot
+    version rebuilt the index 29,265 times for 10,149 links and was no faster than the scan it
+    replaced. The cap keeps this from growing without bound if a caller cycles through catalogs.
+    """
+
+    #: Enough for the two mappings a build alternates between, with room to spare. Small on purpose:
+    #: this holds catalogs alive, so it must not become an unbounded retainer.
+    _MAX_SLOTS = 4
+
+    def __init__(self) -> None:
+        # Insertion-ordered, so the oldest entry is the one evicted.
+        self._slots: dict[int, tuple[dict[str, list[str]], dict[str, list[str]]]] = {}
+
+    def for_catalog(self, catalog: dict[str, list[str]]) -> dict[str, list[str]]:
+        cached = self._slots.get(id(catalog))
+        if cached is not None and cached[0] is catalog:
+            return cached[1]
+
+        buckets: dict[str, list[str]] = {}
+        for paths in catalog.values():
+            for path in paths:
+                buckets.setdefault(_fold_case(path.rsplit("/", 1)[-1]), []).append(path)
+
+        if len(self._slots) >= self._MAX_SLOTS:
+            del self._slots[next(iter(self._slots))]
+        self._slots[id(catalog)] = (catalog, buckets)
+        return buckets
+
+
+def _suffix_matches(suffix: str, catalog: dict[str, list[str]], index: _FilenameIndex) -> list[str]:
+    """Every rel_path that ``suffix`` (a folded ``folder/note.md``) names, in flatten order."""
+    bucket = index.for_catalog(catalog).get(suffix.rsplit("/", 1)[-1], ())
+    return [path for path in bucket if _matches_path_suffix(path, suffix)]
+
+
+#: Shared by ``candidates_for``, which takes a catalog rather than a resolver and so cannot reach
+#: the resolver's own index. One index per process is enough — both callers see the same catalog
+#: object for the length of a build, and a different catalog simply re-keys it.
+_CANDIDATE_INDEX = _FilenameIndex()
+
+
 class NormalizeResolver:
     """Resolves wikilink displays via normalized basename, with path-suffix fallback."""
 
     def __init__(self) -> None:
-        # Cache the flattened path list per catalog identity. catalog is invariant for a whole
-        # VaultGraph.build(), so folder-style links reuse one flatten instead of rebuilding it
-        # on every call. Keyed by id() and single-slot: a new catalog evicts the previous.
-        self._flat_cache_id: int | None = None
-        self._flat_cache: list[str] | None = None
-
-    @staticmethod
-    def _compute_flat(catalog: dict[str, list[str]]) -> list[str]:
-        return [p for paths in catalog.values() for p in paths]
-
-    def _flatten_paths(self, catalog: dict[str, list[str]]) -> list[str]:
-        if self._flat_cache_id != id(catalog) or self._flat_cache is None:
-            self._flat_cache = self._compute_flat(catalog)
-            self._flat_cache_id = id(catalog)
-        return self._flat_cache
+        self._index = _FilenameIndex()
 
     def resolve(self, display: str, catalog: dict[str, list[str]]) -> str | None:
         # Alias ("Note|alias"), anchor ("Note#Section") and an explicit ".md" extension all name
@@ -244,10 +288,11 @@ class NormalizeResolver:
         display = _strip_display(display)
 
         if "/" in display:
-            # Path-suffix resolution: find unique rel_path ending with "display.md"
+            # Path-suffix resolution: find unique rel_path ending with "display.md". Note that an
+            # exact whole-path hit cannot short-circuit — "work/Tasks" names both "work/Tasks.md"
+            # and "a/work/Tasks.md", and the honest answer there is to decline.
             suffix = _fold_case(display) + ".md"
-            all_paths = self._flatten_paths(catalog)
-            matches = [p for p in all_paths if _matches_path_suffix(p, suffix)]
+            matches = _suffix_matches(suffix, catalog, self._index)
             return matches[0] if len(matches) == 1 else None
 
         # Bare-link resolution: normalize and look up in catalog

{graphmark-0.8.0 → graphmark-0.8.1}/tests/test_graph.py

@@ -91,19 +91,15 @@ class TestNormalizeResolver:
         assert self.resolver.resolve("brain/alpha", catalog) == "brain/alpha.md"
         assert self.resolver.resolve("work/gamma", catalog) == "work/gamma.md"
 
-    def test_flatten_runs_once_per_catalog(self, monkeypatch):
+    def test_the_path_index_is_built_once_per_catalog(self):
+        # Was written against the flatten this replaced (#157); the property is the same one and
+        # still worth holding — the O(n) precompute must not run per link.
         catalog = self._catalog("brain/alpha.md", "personal/beta.md")
-        calls = {"n": 0}
-        real_compute = NormalizeResolver._compute_flat
-
-        def counting_compute(cat):
-            calls["n"] += 1
-            return real_compute(cat)
-
-        monkeypatch.setattr(self.resolver, "_compute_flat", counting_compute)
+        self.resolver.resolve("brain/alpha", catalog)
+        buckets = self.resolver._index.for_catalog(catalog)
         for _ in range(5):
             self.resolver.resolve("brain/alpha", catalog)
-        assert calls["n"] == 1  # the O(n) flatten runs once, reused for the rest
+        assert self.resolver._index.for_catalog(catalog) is buckets
 
     def test_new_catalog_triggers_reflatten(self):
         cat1 = self._catalog("brain/alpha.md")

graphmark-0.8.1/tests/test_path_index.py

@@ -0,0 +1,153 @@
+"""Path-suffix resolution is indexed, not scanned (#157).
+
+Every path-qualified display used to flatten the catalog and compare against **every** rel_path.
+With wikilinks that branch is rare — most displays are bare names hitting a dict — so the `O(L·N)`
+ceiling was documented and accepted. #152 changed which branch is hot: every markdown link is
+path-qualified by construction, so the rare branch became the only branch. Measured on a 1120-note
+vault with 10,149 markdown links: 2.9 s against 0.3 s for the same vault read as wikilinks.
+
+Any legal suffix match must end with the display's final component, so bucketing paths by folded
+filename is **exactly** equivalent to scanning them all — same matches, same order, same verdicts.
+This file exists to hold that equivalence, since a pure speedup that changes one answer is a bug.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from graphmark.config import VaultConfig
+from graphmark.graph import NormalizeResolver, VaultGraph, candidates_for
+from graphmark.parse import WikilinkExtractor
+
+
+def _write(root: Path, rel: str, text: str = "") -> None:
+    path = root / rel
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(text, encoding="utf-8")
+
+
+def _build(root: Path, **kw) -> VaultGraph:
+    return VaultGraph.build(VaultConfig(root=root, **kw), WikilinkExtractor(), NormalizeResolver())
+
+
+def _scan_resolve(display: str, catalog: dict[str, list[str]]) -> list[str]:
+    """The pre-#157 behavior, written out: compare against every path, in flatten order.
+
+    Kept as an independent reference rather than by calling the implementation, so the equivalence
+    tests below compare two things instead of one thing with itself.
+    """
+    from graphmark.graph import _fold_case, _matches_path_suffix, _strip_display
+
+    target = _strip_display(display)
+    suffix = _fold_case(target) + ".md"
+    return [p for paths in catalog.values() for p in paths if _matches_path_suffix(p, suffix)]
+
+
+class TestEquivalence:
+    """Same answers as the scan, on every shape that has ever mattered here."""
+
+    def test_matches_the_scan_across_the_shapes_136_turned_on(self, tmp_path):
+        for rel in (
+            "work/Tasks.md",
+            "homework/Tasks.md",
+            "a/b/work/Tasks.md",
+            "docs/Tasks.md",
+            "api-docs/Tasks.md",
+            "Tasks.md",
+            "work/Other.md",
+        ):
+            _write(tmp_path, rel)
+        catalog = _build(tmp_path).catalog
+        resolver = NormalizeResolver()
+        for display in (
+            "work/Tasks",
+            "homework/Tasks",
+            "b/work/Tasks",
+            "docs/Tasks",
+            "Tasks",
+            "work/Other",
+            "nope/Tasks",
+            "work/Nope",
+        ):
+            expected = sorted(_scan_resolve(display, catalog))
+            assert sorted(candidates_for(display, catalog)) == expected, display
+            if "/" in display:
+                unique = expected[0] if len(expected) == 1 else None
+                assert resolver.resolve(display, catalog) == unique, display
+
+    def test_case_and_unicode_form_still_fold(self, tmp_path):
+        _write(tmp_path, "Work/Tasks.md")
+        catalog = _build(tmp_path).catalog
+        assert NormalizeResolver().resolve("work/tasks", catalog) == "Work/Tasks.md"
+
+    def test_a_genuine_collision_is_still_declined(self, tmp_path):
+        _write(tmp_path, "a/work/Tasks.md")
+        _write(tmp_path, "b/work/Tasks.md")
+        catalog = _build(tmp_path).catalog
+        assert NormalizeResolver().resolve("work/Tasks", catalog) is None
+        assert candidates_for("work/Tasks", catalog) == ["a/work/Tasks.md", "b/work/Tasks.md"]
+
+    def test_a_whole_path_and_a_deeper_suffix_both_match(self, tmp_path):
+        # The case that forbids short-circuiting on an exact hit: "work/Tasks" names both, so the
+        # resolver must decline. An index that returned the exact match would change the verdict.
+        _write(tmp_path, "work/Tasks.md")
+        _write(tmp_path, "a/work/Tasks.md")
+        catalog = _build(tmp_path).catalog
+        assert NormalizeResolver().resolve("work/Tasks", catalog) is None
+        assert candidates_for("work/Tasks", catalog) == ["a/work/Tasks.md", "work/Tasks.md"]
+
+    def test_a_degenerate_trailing_slash_display(self, tmp_path):
+        _write(tmp_path, "a/b.md")
+        catalog = _build(tmp_path).catalog
+        assert candidates_for("a/", catalog) == _scan_resolve("a/", catalog)
+
+
+class TestCacheSafety:
+    def test_two_catalogs_do_not_share_an_index(self, tmp_path):
+        _write(tmp_path, "one/note.md")
+        first = _build(tmp_path).catalog
+        _write(tmp_path, "two/other.md")
+        second = _build(tmp_path).catalog
+        resolver = NormalizeResolver()
+        assert resolver.resolve("one/note", first) == "one/note.md"
+        assert resolver.resolve("two/other", second) == "two/other.md"
+        # Back to the first catalog: a single-slot cache must re-key, not answer from the second.
+        assert resolver.resolve("two/other", first) is None
+        assert resolver.resolve("one/note", first) == "one/note.md"
+
+    def test_alternating_between_two_catalogs_does_not_rebuild(self):
+        """The regression guard for what actually made the first attempt useless.
+
+        `_diagnose` consults the in-scope catalog and the out-of-scope one for the same
+        display, so a single-slot cache alternates and misses every time. That version rebuilt
+        the index 29,265 times for 10,149 links and was no faster than the scan it replaced — a
+        "speedup" that was entirely cache thrash. The buckets' identity is the observable.
+        """
+        from graphmark.graph import _FilenameIndex
+
+        a = {"one": ["one/note.md"]}
+        b = {"two": ["two/other.md"]}
+        index = _FilenameIndex()
+        first_a, first_b = index.for_catalog(a), index.for_catalog(b)
+        for _ in range(5):
+            assert index.for_catalog(a) is first_a
+            assert index.for_catalog(b) is first_b
+
+    def test_the_slot_count_is_bounded(self):
+        # The cache holds catalogs alive, so it must not become an unbounded retainer.
+        from graphmark.graph import _FilenameIndex
+
+        index = _FilenameIndex()
+        for i in range(_FilenameIndex._MAX_SLOTS + 3):
+            index.for_catalog({f"k{i}": [f"f{i}/n.md"]})
+        assert len(index._slots) <= _FilenameIndex._MAX_SLOTS
+
+    def test_a_mutated_catalog_is_not_answered_from_a_stale_index(self, tmp_path):
+        _write(tmp_path, "one/note.md")
+        catalog = _build(tmp_path).catalog
+        resolver = NormalizeResolver()
+        assert resolver.resolve("one/note", catalog) == "one/note.md"
+        # Same object, new content. The cache keys on identity, so this is its honest limit:
+        # callers must not mutate a catalog mid-build. Asserted so the limit is visible.
+        catalog.setdefault("late", []).append("late/late.md")
+        assert resolver.resolve("late/late", catalog) is None