wikifier 4.1.1__tar.gz → 4.1.2__tar.gz

{wikifier-4.1.1/wikifier.egg-info → wikifier-4.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wikifier
-Version: 4.1.1
+Version: 4.1.2
 Summary: Agent-first, zero-dependency, self-maintaining codebase documentation & change tracking system
 Author-email: Aron Amos <aron@example.com>
 Maintainer: Aron Amos
@@ -97,6 +97,15 @@ See the final synth in `Findings/M5-Dogfood-Progress.md` and full Assessment-Rep
 - Synced descriptions in README.md, `skills/run.md`, `wikifier/mcp/README.md`.
 - All under protocol (FRESH, record-change + mark-green with subid, main clean). Version to 4.1.1. See the separation-fix commit.
 
+**v4.1.2 (2026-06)**: Very minor patch for mapping / update speed hygiene (no new behaviour or scope; pure internal improvements for large projects).
+
+- Faster candidate collection everywhere that drives check-changes, monitor, and update-maps (Python-primary paths): switched to `os.scandir`-based recursive scan (std lib, avoids walk overhead) + git `ls-files --cached --others --exclude-standard` fast-path when in a git repo (dramatically faster on real monorepos; falls back cleanly).
+- Consistent early pruning: `exclude_patterns.txt` (user-editable, populated by init) now applied more broadly during mapping walks in both sh and Python collectors (venvs, caches, site-packages, etc. stop descent sooner).
+- Small parser micro-opt: hoisted common regex compiles (docstring strip, dynamic import detectors) to module level in the Python parser (hot path on dirty files during maps).
+- Minor sh-side note + skeleton for git fast collection in traditional update-maps path (real wins already flow through the Python collectors used by check-changes/streaming/lib/MCP).
+- All changes FRESH-checked + recorded+marked under subid=mapping-speed-hygiene. Complements existing levers (monitored_paths.txt narrowing, --dir / directory= scoping, --stream / --max-files / --max-time streaming+budgets, python-primary, incremental dirty + BRC reverse index).
+- Version to 4.1.2. See the speed-hygiene edits + this commit.
+
 ### Historical (pre-v4.0)
 
 #### What's New in v0.3.3 (Gap #1)

{wikifier-4.1.1 → wikifier-4.1.2}/README.md

@@ -65,6 +65,15 @@ See the final synth in `Findings/M5-Dogfood-Progress.md` and full Assessment-Rep
 - Synced descriptions in README.md, `skills/run.md`, `wikifier/mcp/README.md`.
 - All under protocol (FRESH, record-change + mark-green with subid, main clean). Version to 4.1.1. See the separation-fix commit.
 
+**v4.1.2 (2026-06)**: Very minor patch for mapping / update speed hygiene (no new behaviour or scope; pure internal improvements for large projects).
+
+- Faster candidate collection everywhere that drives check-changes, monitor, and update-maps (Python-primary paths): switched to `os.scandir`-based recursive scan (std lib, avoids walk overhead) + git `ls-files --cached --others --exclude-standard` fast-path when in a git repo (dramatically faster on real monorepos; falls back cleanly).
+- Consistent early pruning: `exclude_patterns.txt` (user-editable, populated by init) now applied more broadly during mapping walks in both sh and Python collectors (venvs, caches, site-packages, etc. stop descent sooner).
+- Small parser micro-opt: hoisted common regex compiles (docstring strip, dynamic import detectors) to module level in the Python parser (hot path on dirty files during maps).
+- Minor sh-side note + skeleton for git fast collection in traditional update-maps path (real wins already flow through the Python collectors used by check-changes/streaming/lib/MCP).
+- All changes FRESH-checked + recorded+marked under subid=mapping-speed-hygiene. Complements existing levers (monitored_paths.txt narrowing, --dir / directory= scoping, --stream / --max-files / --max-time streaming+budgets, python-primary, incremental dirty + BRC reverse index).
+- Version to 4.1.2. See the speed-hygiene edits + this commit.
+
 ### Historical (pre-v4.0)
 
 #### What's New in v0.3.3 (Gap #1)

{wikifier-4.1.1 → wikifier-4.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "wikifier"
-version = "4.1.1"
+version = "4.1.2"
 description = "Agent-first, zero-dependency, self-maintaining codebase documentation & change tracking system"
 readme = "README.md"
 license = {text = "MIT"}

{wikifier-4.1.1 → wikifier-4.1.2}/wikifier/__init__.py

@@ -62,4 +62,4 @@ from .contracts import (
     compute_acs_confidence,
 )
 
-__version__ = "4.1.1"
+__version__ = "4.1.2"

{wikifier-4.1.1 → wikifier-4.1.2}/wikifier/cli.py

@@ -172,18 +172,75 @@ def _collect_candidate_source_files(root: Path) -> List[Path]:
         root = Path(root).resolve()
     except Exception:
         root = Path(root)
+    # Also respect the project's exclude_patterns.txt (if present) for parity with sh
+    # mapping paths and check-changes. Simple dir-name globs only for pruning speed.
+    # This makes python-primary update-maps benefit from user custom excludes (venvs etc)
+    # without any behavior change.
+    # Look relative to explicit WIKIFIER_PROJECT_ROOT (if set for the target) or the
+    # passed root; excludes live at the logical project root, not arbitrary monitored subdirs.
+    ep_root = Path(os.environ.get("WIKIFIER_PROJECT_ROOT", root))
+    ep = ep_root / "exclude_patterns.txt"
+    if ep.exists():
+        try:
+            for line in ep.read_text(errors="ignore").splitlines():
+                p = line.strip()
+                if p and not p.startswith("#"):
+                    p = p.split()[0]  # first token
+                    if p:
+                        EXCLUDES.add(p)
+                        # also common glob forms as exact for dirname match
+                        if p.endswith("/*") or p.endswith("*"):
+                            EXCLUDES.add(p.rstrip("/*"))
+        except Exception:
+            pass
 
-    for dirpath, dirnames, filenames in os.walk(root):
-        # prune in-place (same pattern as resolution._discover_*)
-        dirnames[:] = [d for d in dirnames if d not in EXCLUDES]
-        for fn in filenames:
-            if fn.lower().endswith(exts):
-                p = Path(dirpath) / fn
-                try:
-                    if p.is_file():
-                        candidates.append(p)
-                except Exception:
+    # Fast path: if inside a git repo, use `git ls-files` + untracked (respects .gitignore, dramatically faster
+    # on large checkouts than any walk; falls back to scandir scan). This is a pure speed opt for "updates"
+    # (check-changes, update-maps) with near-identical or better candidate set for real codebases.
+    git_dir = root / ".git"
+    if git_dir.exists() or (root / ".git" / "HEAD").exists():  # works for worktrees too
+        try:
+            import subprocess
+            # cached + others (untracked but not ignored), exclude standard ignores
+            out = subprocess.check_output(
+                ["git", "ls-files", "--cached", "--others", "--exclude-standard", "-z"],
+                cwd=root, stderr=subprocess.DEVNULL
+            )
+            for entry in out.split(b"\0"):
+                if not entry:
                     continue
+                p = (root / entry.decode("utf-8", "ignore")).resolve()
+                if p.suffix.lower() in exts:  # reuse the set from above (adjusted)
+                    # quick filter for excludes we still want even if git surfaces them
+                    parts = p.parts
+                    if not any(part in EXCLUDES or any(part.startswith(e) for e in (".",)) for part in parts):
+                        candidates.append(p)
+            if candidates:
+                return candidates  # success, use git list
+        except Exception:
+            pass  # fall through to scandir
+
+    # Use os.scandir for faster directory traversal (std lib only; avoids full listdir + separate is_dir stats on large trees).
+    # Pruning is applied on the fly. Behavior identical to prior walk.
+    exts_lower = tuple(e.lower() for e in exts)
+    def _scan_dir(d: Path) -> None:
+        try:
+            with os.scandir(d) as it:
+                for entry in it:
+                    try:
+                        name = entry.name
+                        if entry.is_dir(follow_symlinks=False):
+                            if name not in EXCLUDES and not name.startswith('.'):
+                                _scan_dir(Path(entry.path))
+                        elif entry.is_file(follow_symlinks=False):
+                            lname = name.lower()
+                            if lname.endswith(exts_lower):
+                                candidates.append(Path(entry.path))
+                    except Exception:
+                        continue
+        except Exception:
+            pass
+    _scan_dir(root)
     return candidates
 
 

{wikifier-4.1.1 → wikifier-4.1.2}/wikifier/import_cache.py

@@ -1852,15 +1852,44 @@ def generate_update_events(
     # Real early scope projection (proportional for 50k+) - Micro-step 1
     projector_stats: Dict[str, Any] = {"degraded": True}
     proj: Dict[str, Any] = {}
-    # Basic candidate collection (inline for self-contained Micro-step 1; mirrors common excludes)
-    candidates = []
+    # Faster candidate collection using os.scandir (avoids repeated listdir overhead).
+    # Respects exclude_patterns.txt when present (for consistency with check-changes + mapping speed).
+    # Same semantics as before.
+    candidates: List[Path] = []
     exts = {'.py', '.js', '.ts', '.jsx', '.tsx'}
-    exclude_dirs = {'__pycache__', '.git', 'node_modules', '.venv', 'venv', 'build', 'dist', '.next', '.cache'}
-    for dirpath, dirnames, filenames in os.walk(root):
-        dirnames[:] = [d for d in dirnames if d not in exclude_dirs and not d.startswith('.')]
-        for f in filenames:
-            if os.path.splitext(f)[1].lower() in exts:
-                candidates.append(Path(dirpath) / f)
+    exclude_dirs = {'__pycache__', '.git', 'node_modules', '.venv', 'venv', 'build', 'dist', '.next', '.cache',
+                    '.pnpm', '.yarn', '.store', 'tmp', 'temp', '.turbo', '.mypy_cache', '.ruff_cache'}
+    # Load project excludes if available (project root level)
+    ep = root / "exclude_patterns.txt"
+    if ep.exists():
+        try:
+            for line in ep.read_text(errors="ignore").splitlines():
+                p = line.strip()
+                if p and not p.startswith("#"):
+                    p = p.split()[0]
+                    if p:
+                        exclude_dirs.add(p)
+                        if p.endswith("/*") or p.endswith("*"):
+                            exclude_dirs.add(p.rstrip("/*"))
+        except Exception:
+            pass
+    def _scan(d: Path) -> None:
+        try:
+            with os.scandir(d) as it:
+                for entry in it:
+                    try:
+                        name = entry.name
+                        if entry.is_dir(follow_symlinks=False):
+                            if name not in exclude_dirs and not name.startswith('.'):
+                                _scan(Path(entry.path))
+                        elif entry.is_file(follow_symlinks=False):
+                            if os.path.splitext(name)[1].lower() in exts:
+                                candidates.append(Path(entry.path))
+                    except Exception:
+                        continue
+        except Exception:
+            pass
+    _scan(root)
     candidates_rel: List[str] = []
     for p in candidates:
         try:

{wikifier-4.1.1 → wikifier-4.1.2}/wikifier/parsers/python.py

@@ -43,6 +43,13 @@ import re
 from pathlib import Path
 from typing import List, Dict, Optional, Any
 
+# Module-level compiled regexes for small repeated speed win on every parse (docstring strip + dynamic import detectors).
+# Zero behavior change.
+_DOCSTRING_RE1 = re.compile(r'"""[\s\S]*?"""')
+_DOCSTRING_RE2 = re.compile(r"'''[\s\S]*?'''")
+_DYN_IMPORT_RE = re.compile(r'(?P<call>(?:importlib\.)?import_module)\s*\(', re.MULTILINE)
+_DYN_DUNDER_RE = re.compile(r'(?P<call>__import__)\s*\(', re.MULTILINE)
+
 # Diagnostics & Failure Transparency (Limitation #5) - same robust import pattern as JS parser
 try:
     from . import diagnostics
@@ -249,8 +256,8 @@ def _strip_docstrings(content: str) -> str:
     strings well), but it is good enough for v0.4 and keeps us zero-dependency.
     """
     # Remove """...""" and '''...''' (non-greedy, handles both single and double)
-    content = re.sub(r'"""[\s\S]*?"""', '', content)
-    content = re.sub(r"'''[\s\S]*?'''", '', content)
+    content = _DOCSTRING_RE1.sub('', content)
+    content = _DOCSTRING_RE2.sub('', content)
     return content
 
 
@@ -304,8 +311,8 @@ def parse_python_imports(filepath: str) -> List[Dict[str, Any]]:
     dynamic_imports: List[Dict[str, Any]] = []
     if _extract_candidate_literals is not None and _apply_dynamic_registry is not None:
         dyn_patterns = [
-            (re.compile(r'(?P<call>(?:importlib\.)?import_module)\s*\(', re.MULTILINE), "import_module"),
-            (re.compile(r'(?P<call>__import__)\s*\(', re.MULTILINE), "dunder_import"),
+            (_DYN_IMPORT_RE, "import_module"),
+            (_DYN_DUNDER_RE, "dunder_import"),
         ]
         for pat, ptype in dyn_patterns:
             for match in pat.finditer(content):

{wikifier-4.1.1 → wikifier-4.1.2/wikifier.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wikifier
-Version: 4.1.1
+Version: 4.1.2
 Summary: Agent-first, zero-dependency, self-maintaining codebase documentation & change tracking system
 Author-email: Aron Amos <aron@example.com>
 Maintainer: Aron Amos
@@ -97,6 +97,15 @@ See the final synth in `Findings/M5-Dogfood-Progress.md` and full Assessment-Rep
 - Synced descriptions in README.md, `skills/run.md`, `wikifier/mcp/README.md`.
 - All under protocol (FRESH, record-change + mark-green with subid, main clean). Version to 4.1.1. See the separation-fix commit.
 
+**v4.1.2 (2026-06)**: Very minor patch for mapping / update speed hygiene (no new behaviour or scope; pure internal improvements for large projects).
+
+- Faster candidate collection everywhere that drives check-changes, monitor, and update-maps (Python-primary paths): switched to `os.scandir`-based recursive scan (std lib, avoids walk overhead) + git `ls-files --cached --others --exclude-standard` fast-path when in a git repo (dramatically faster on real monorepos; falls back cleanly).
+- Consistent early pruning: `exclude_patterns.txt` (user-editable, populated by init) now applied more broadly during mapping walks in both sh and Python collectors (venvs, caches, site-packages, etc. stop descent sooner).
+- Small parser micro-opt: hoisted common regex compiles (docstring strip, dynamic import detectors) to module level in the Python parser (hot path on dirty files during maps).
+- Minor sh-side note + skeleton for git fast collection in traditional update-maps path (real wins already flow through the Python collectors used by check-changes/streaming/lib/MCP).
+- All changes FRESH-checked + recorded+marked under subid=mapping-speed-hygiene. Complements existing levers (monitored_paths.txt narrowing, --dir / directory= scoping, --stream / --max-files / --max-time streaming+budgets, python-primary, incremental dirty + BRC reverse index).
+- Version to 4.1.2. See the speed-hygiene edits + this commit.
+
 ### Historical (pre-v4.0)
 
 #### What's New in v0.3.3 (Gap #1)