sourcecode 2.5.15__tar.gz → 2.5.16__tar.gz

{sourcecode-2.5.15 → sourcecode-2.5.16}/CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## [2.5.16] — 2026-07-19
+
+**Performance campaign close-out: one real algorithmic win, cost-per-symbol
+measurable, one candidate refuted by measurement.** The ADR-0006 harness localized
+cold wall to the `analysis` phase (90–96% on every non-trivial repo); profiling
+inside it found — and this release fixes — the one structural bottleneck, populates
+the last empty baseline axes, and records a negative result that stopped a
+non-winning change from landing.
+
+### Performance
+
+- **Killed an O(files × total_symbols) scan in per-file Java contract projection**
+  (`ast_extractor.py`, `context_graph.py`): `_java_contract_from_graph` ran once per
+  Java file and called the whole-repo `graph.symbols()`/`relations()` unfiltered, then
+  kept only its own file's rows. Added lazy once-built groupings
+  `ContextGraph.symbols_in_file(path)` / `relations_from(fqn)` (O(1) after build); the
+  projection now walks only edges sourced at its own types. Behavior-equivalent
+  (normalized old-vs-new on petclinic + keycloak; 1430 IR/spring/endpoint/impact tests
+  green). Cold wall p50, same host, speedup tracking file count as the O(files × N)
+  model predicts: neo4j 33.6→16.7s (2.0×), broadleaf 27.9→15.4s (1.8×),
+  keycloak 20.8→14.1s (1.5×), openmrs 7.9→5.8s (1.4×), ofbiz 19.4→15.0s (1.3×).
+
+### Added
+
+- **`repo.classes` / `repo.methods` populated in the perf baseline** from repo-ir's
+  symbol taxonomy (`repository_ir.py`, `scripts/perf_harness.py`): `analysis_meta` now
+  emits a deterministic `symbol_kinds` breakdown plus two rollups —
+  `classes` = class+interface+enum+annotation+record, `methods` = method+constructor —
+  and the harness reads them via one `repo-ir --summary-only` pass off the measured path
+  (never in `wall_ms`; repo-ir failure falls back to 0, an honest "unmeasured"). Closes
+  the last empty axes in the `perf-baseline-v1` schema, enabling cost-per-symbol.
+
+### Notes
+
+- **A body-fact-scanning optimization was investigated and reverted after measurement
+  showed no win.** cProfile flagged per-method body re-slicing (`_method_body` 44508× on
+  ofbiz) as a lead; a byte-identical single-pass rewrite (0 mismatches / 5555 files)
+  measured 7674→7721 ms — no improvement. The profiler had flagged a call-count-heavy but
+  wall-cheap region; the real cost is inherent regex/char scanning, already past its
+  algorithmic win. The change was dropped rather than accrue complexity on a hot,
+  output-critical path for no gain.
+
 ## [2.5.15] — 2026-07-19
 
 **Bootstrap detection recovers the real entry point on multi-`main` repos, without

{sourcecode-2.5.15 → sourcecode-2.5.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sourcecode
-Version: 2.5.15
+Version: 2.5.16
 Summary: Persistent structural context and ultra-fast repeated analysis for AI coding agents
 License-File: LICENSE
 Keywords: agents,ai,codebase,context,developer-tools,llm

{sourcecode-2.5.15 → sourcecode-2.5.16}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "sourcecode"
-version = "2.5.15"
+version = "2.5.16"
 description = "Persistent structural context and ultra-fast repeated analysis for AI coding agents"
 readme = "README.md"
 requires-python = ">=3.9"

sourcecode-2.5.16/scripts/perf_harness.py

@@ -0,0 +1,252 @@
+#!/usr/bin/env python3
+"""Performance harness — runs the reference fleet under the ADR-0006 contract.
+
+For every (command x repo x mode) cell it runs the CLI N times, records
+wall-clock, peak RSS and (when the in-process recorder is armed) per-phase
+timings, then writes one ``perf-baseline-v1`` JSON artifact.
+
+Methodology is fixed by ADR-0006 §5: cold and warm are separate cells and never
+averaged; the first warm run is discarded as cache fill; central tendency is the
+median; p95 is always reported; cross-machine numbers are recorded, never diffed.
+
+Usage:
+    python3 scripts/perf_harness.py --quick              # petclinic only, 2 runs
+    python3 scripts/perf_harness.py                      # full fleet, 7 runs
+    python3 scripts/perf_harness.py --repos keycloak eureka --runs 5
+    python3 scripts/perf_harness.py --dry-run            # print the plan, run nothing
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import resource
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+# Import the frozen collector from the package under test.
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
+from sourcecode import __version__ as TOOL_VERSION  # noqa: E402
+from sourcecode import perf  # noqa: E402
+
+# ── Fleet definition ────────────────────────────────────────────────────────
+# Directory name under the fleet root. Repos are pinned by their checked-out
+# commit at run time (recorded in each artifact), so a size change is never
+# mistaken for a perf change.
+DEFAULT_FLEET_ROOT = Path(
+    os.environ.get("PERF_FLEET_ROOT", "/Users/user/Documents/workspace/testing")
+)
+FLEET: dict[str, str] = {
+    "petclinic": "spring-petclinic",
+    "broadleaf": "BroadleafCommerce",
+    "openmrs": "openmrs-core",
+    "keycloak": "keycloak",
+    "eureka": "eureka",
+    "jobrunr": "jobrunr",
+    "ofbiz": "ofbiz-framework",
+    "alfresco": "alfresco-community-repo",
+    "jenkins": "jenkins",
+    "neo4j": "neo4j",
+}
+
+# Commands to profile. The empty string is the root analysis (`ask <repo>`);
+# named entries are subcommands (`ask <cmd> <repo>`).
+DEFAULT_COMMANDS = [""]
+
+ASK_BIN = os.environ.get("ASK_BIN", "ask")
+BYTES_PER_KB = 1024
+# darwin ru_maxrss is bytes; linux reports kilobytes.
+RSS_UNIT = 1 if sys.platform == "darwin" else BYTES_PER_KB
+
+
+def git_commit(repo_path: Path) -> str:
+    try:
+        out = subprocess.run(
+            ["git", "-C", str(repo_path), "rev-parse", "--short", "HEAD"],
+            capture_output=True, text=True, timeout=15,
+        )
+        return out.stdout.strip() or "unknown"
+    except (subprocess.SubprocessError, OSError):
+        return "unknown"
+
+
+def repo_size(repo_path: Path) -> dict[str, int]:
+    """Repo scaling axes for cost-per-unit analysis.
+
+    ``files`` / ``bytes`` are a cheap rglob over every ``*.java`` (the frozen file
+    axis — includes tests). ``classes`` / ``methods`` come from repo-ir's
+    ``analysis_meta`` rollup (the IR owns the symbol taxonomy). This is one
+    ``repo-ir --summary-only`` pass off the measured path; it never touches
+    ``wall_ms``. If repo-ir fails the symbol counts fall back to 0 — an honest
+    "unmeasured", never a fabricated number.
+    """
+    files = 0
+    total = 0
+    for p in repo_path.rglob("*.java"):
+        try:
+            total += p.stat().st_size
+            files += 1
+        except OSError:
+            continue
+    classes, methods = _repo_symbol_counts(repo_path)
+    return {"files": files, "bytes": total, "classes": classes, "methods": methods}
+
+
+def _repo_symbol_counts(repo_path: Path) -> tuple[int, int]:
+    """(classes, methods) from ``repo-ir --summary-only``; (0, 0) on any failure."""
+    with tempfile.TemporaryDirectory(prefix="perf-ir-") as td:
+        out = Path(td) / "ir.json"
+        try:
+            subprocess.run(
+                [ASK_BIN, "repo-ir", str(repo_path), "--summary-only", "-o", str(out)],
+                capture_output=True, timeout=1800, check=False,
+            )
+            meta = json.loads(out.read_text(encoding="utf-8")).get("analysis_meta", {})
+            return int(meta.get("classes", 0) or 0), int(meta.get("methods", 0) or 0)
+        except (subprocess.SubprocessError, OSError, ValueError):
+            return 0, 0
+
+
+def build_argv(command: str, repo_path: Path, cold: bool) -> list[str]:
+    argv = [ASK_BIN]
+    if command:
+        argv.append(command)
+    argv.append(str(repo_path))
+    argv += ["--format", "json"]
+    if cold:
+        argv.append("--no-cache")
+    return argv
+
+
+def run_once(argv: list[str], env: dict[str, str], perf_log: Path) -> tuple[float, float, dict | None]:
+    """Run the CLI once. Returns (wall_ms, rss_bytes, phase_ms | None)."""
+    if perf_log.exists():
+        perf_log.unlink()
+    rss_before = resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss
+    t0 = time.perf_counter()
+    subprocess.run(argv, env=env, capture_output=True, timeout=1800)
+    wall_ms = (time.perf_counter() - t0) * 1000.0
+    rss_after = resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss
+    # ru_maxrss for RUSAGE_CHILDREN is the peak among reaped children; the delta
+    # approximates this run's peak. Approximate by construction — documented.
+    rss_bytes = max(rss_after - rss_before, rss_after) * RSS_UNIT if rss_after else 0.0
+    phase_ms = _read_phase_log(perf_log)
+    return wall_ms, float(rss_bytes), phase_ms
+
+
+def _read_phase_log(perf_log: Path) -> dict | None:
+    if not perf_log.exists():
+        return None
+    try:
+        last = perf_log.read_text(encoding="utf-8").strip().splitlines()[-1]
+        return json.loads(last).get("phase_ms") or None
+    except (OSError, IndexError, ValueError):
+        return None
+
+
+def measure_cell(command: str, repo_path: Path, mode: str, runs: int) -> dict:
+    cold = mode == "cold"
+    walls: list[float] = []
+    rsss: list[float] = []
+    phases: list[dict] = []
+
+    with tempfile.TemporaryDirectory(prefix="perf-cache-") as cache_dir, \
+         tempfile.TemporaryDirectory(prefix="perf-log-") as log_dir:
+        perf_log = Path(log_dir) / "phases.jsonl"
+        base_env = dict(os.environ)
+        base_env[perf.ENV_ENABLE] = "1"
+        base_env[perf.ENV_LOG] = str(perf_log)
+        base_env["SOURCECODE_CACHE_DIR"] = cache_dir
+
+        # Warm mode: one un-recorded priming run to populate the cache, then the
+        # first measured warm run is still discarded (belt and suspenders).
+        n = runs + 1 if mode == "warm" else runs
+        for i in range(n):
+            argv = build_argv(command, repo_path, cold=cold)
+            if cold:
+                # Each cold run gets a pristine cache dir.
+                for child in Path(cache_dir).iterdir():
+                    shutil.rmtree(child, ignore_errors=True) if child.is_dir() else child.unlink()
+            wall, rss, phase = run_once(argv, base_env, perf_log)
+            if mode == "warm" and i == 0:
+                continue  # discard cache-fill run
+            walls.append(wall)
+            rsss.append(rss)
+            if phase:
+                phases.append(phase)
+
+    return perf.build_baseline_cell(
+        tool_version=TOOL_VERSION,
+        command=command or "ask",
+        repo={"name": repo_path.name, "commit": git_commit(repo_path), **repo_size(repo_path)},
+        mode=mode,
+        wall_ms_runs=walls,
+        rss_bytes_runs=rsss,
+        phase_ms_runs=phases or None,
+    )
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--repos", nargs="*", help="Subset of fleet names (default: all).")
+    ap.add_argument("--commands", nargs="*", default=DEFAULT_COMMANDS,
+                    help="Commands to profile; '' = root analysis (default).")
+    ap.add_argument("--modes", nargs="*", default=["cold", "warm"], choices=["cold", "warm"])
+    ap.add_argument("--runs", type=int, default=7, help="Measured runs per cell (>=5 per ADR-0006).")
+    ap.add_argument("--fleet-root", type=Path, default=DEFAULT_FLEET_ROOT)
+    ap.add_argument("--out", type=Path, default=None,
+                    help="Output dir (default: docs/perf/baselines/<tool_version>).")
+    ap.add_argument("--quick", action="store_true", help="petclinic only, 2 runs — smoke test.")
+    ap.add_argument("--dry-run", action="store_true", help="Print the plan, run nothing.")
+    args = ap.parse_args()
+
+    if args.quick:
+        args.repos = ["petclinic"]
+        args.runs = 2
+
+    repos = args.repos or list(FLEET)
+    out_dir = args.out or (Path(__file__).resolve().parent.parent / "docs" / "perf" / "baselines" / TOOL_VERSION)
+
+    plan = [
+        (cmd, name, mode)
+        for cmd in args.commands
+        for name in repos
+        for mode in args.modes
+    ]
+    print(f"perf-harness {TOOL_VERSION} — {len(plan)} cells, {args.runs} runs each")
+    print(f"fleet root: {args.fleet_root}")
+    print(f"output:     {out_dir}")
+    if args.dry_run:
+        for cmd, name, mode in plan:
+            print(f"  [{mode:4}] {name:10} {cmd or 'ask'}")
+        return 0
+
+    out_dir.mkdir(parents=True, exist_ok=True)
+    for cmd, name, mode in plan:
+        subdir = FLEET.get(name, name)
+        repo_path = args.fleet_root / subdir
+        if not repo_path.is_dir():
+            print(f"  SKIP {name}: {repo_path} not found", file=sys.stderr)
+            continue
+        print(f"  [{mode:4}] {name:10} {cmd or 'ask'} ...", end="", flush=True)
+        try:
+            cell = measure_cell(cmd, repo_path, mode, args.runs)
+        except subprocess.TimeoutExpired:
+            print(" TIMEOUT", file=sys.stderr)
+            continue
+        fname = f"{name}__{cmd or 'ask'}__{mode}.json"
+        (out_dir / fname).write_text(json.dumps(cell, indent=2) + "\n", encoding="utf-8")
+        w = cell["wall_ms"]
+        print(f" p50={w['p50']}ms p95={w['p95']}ms -> {fname}")
+
+    print(f"done. artifacts in {out_dir}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{sourcecode-2.5.15 → sourcecode-2.5.16}/src/sourcecode/__init__.py

@@ -4,4 +4,4 @@ ASK Engine is the product. ``ask`` is the canonical CLI command; ``sourcecode``
 the legacy compatibility alias and the Python/PyPI package name. See
 docs/PRODUCT_IDENTITY.md (normative)."""
 
-__version__ = "2.5.15"
+__version__ = "2.5.16"

{sourcecode-2.5.15 → sourcecode-2.5.16}/src/sourcecode/ast_extractor.py

@@ -967,25 +967,26 @@ def _java_contract_from_graph(rel_path: str, graph: "Any") -> FileContract:
     nodes/edges instead of re-parsing source. Method signatures are the graph's
     canonical compact form (`(Type)->Return`); annotations are node-grounded.
     """
-    file_syms = [s for s in graph.symbols() if s.source_file == rel_path]
+    file_syms = graph.symbols_in_file(rel_path)
     type_syms = [s for s in file_syms if s.kind in ("class", "interface", "enum", "annotation")]
     type_fqns = {s.fqn for s in type_syms}
 
     # extends / implements edges out of this file's types (targets are simple
     # names for out-of-repo supertypes, resolved FQNs for in-repo ones — the
-    # same forms the old regex captured for the unresolved case).
+    # same forms the old regex captured for the unresolved case). Only edges
+    # sourced at this file's types matter, so index straight to them instead of
+    # scanning the whole-repo relation set once per file.
     extends_of: dict[str, list[str]] = {}
     implements_of: dict[str, list[str]] = {}
     import_targets: set[str] = set()
-    for r in graph.relations():
-        if r.source not in type_fqns:
-            continue
-        if r.kind == "extends":
-            extends_of.setdefault(r.source, []).append(r.target)
-        elif r.kind == "implements":
-            implements_of.setdefault(r.source, []).append(r.target)
-        elif r.kind == "imports":
-            import_targets.add(r.target)
+    for fqn in type_fqns:
+        for r in graph.relations_from(fqn):
+            if r.kind == "extends":
+                extends_of.setdefault(r.source, []).append(r.target)
+            elif r.kind == "implements":
+                implements_of.setdefault(r.source, []).append(r.target)
+            elif r.kind == "imports":
+                import_targets.add(r.target)
 
     exports: list[ExportRecord] = []
     types: list[TypeDefinition] = []

{sourcecode-2.5.15 → sourcecode-2.5.16}/src/sourcecode/cli.py

@@ -15,6 +15,7 @@ from sourcecode import __version__
 from sourcecode.error_schema import INVALID_INPUT_CODE, build_error_envelope
 from sourcecode.entrypoint_classifier import is_production_entry_point, normalize_entry_point
 from sourcecode.progress import Progress
+from sourcecode import perf
 from sourcecode.caller_metrics import CALLER_METRIC_RECONCILIATION
 from sourcecode.repository_ir import extract_java_endpoints as _extract_java_endpoints
 
@@ -1703,6 +1704,7 @@ def main(
     _progress = Progress()
     _progress.start("scanning files")
 
+    _perf_discovery = perf.start()
     scanner = AdaptiveScanner(target, topology=_topology, base_depth=effective_depth,
                                extra_excludes=_extra_excludes)
     raw_tree = scanner.scan_tree()
@@ -1710,6 +1712,8 @@ def main(
     _progress.update("parsing manifests")
     # 2. Filter .env and *.secret entries from file tree (SEC-02, all levels)
     file_tree = filter_sensitive_files(raw_tree, redactor)
+    perf.stop("discovery", _perf_discovery)
+    _perf_detection = perf.start()
     detector = ProjectDetector(build_default_detectors())
     workspace_analysis = WorkspaceAnalyzer().analyze(target, manifests)
 
@@ -1951,6 +1955,8 @@ def main(
         else None
     )
 
+    perf.stop("detection", _perf_detection)
+    _perf_analysis = perf.start()
     # 3. Build schema
     # Compute analyzer fingerprints: short hashes of each analyzer's key rule
     # constants so that a rule change is always visible in the output, regardless
@@ -2411,8 +2417,10 @@ def main(
                     err=True,
                 )
 
+    perf.stop("analysis", _perf_analysis)
     # 4. Serialize
     _progress.update("serializing output")
+    _perf_serialize = perf.start()
     if _is_contract_mode and not agent:
         from sourcecode.serializer import contract_view as _contract_view
         _depth = _CONTRACT_DEPTH.get(mode, "minimal")
@@ -2455,6 +2463,9 @@ def main(
             raw_dict = redact_dict(raw_dict)
         content = _serialize_dict(raw_dict, format)
 
+    perf.stop("serialize", _perf_serialize)
+    perf.flush_recorder()
+
     # 5. Telemetry (fire-and-forget, never blocks)
     try:
         from sourcecode import telemetry as _tel

{sourcecode-2.5.15 → sourcecode-2.5.16}/src/sourcecode/context_graph.py

@@ -408,6 +408,7 @@ class ContextGraph:
         "_cir", "_nodes_by_fqn", "_nodes", "_build_ms",
         "_body_index", "_literal_index", "_guard_index", "_span_index",
         "_class_type_index", "_relations_all", "_symbols_sorted",
+        "_symbols_by_file", "_relations_by_source",
     )
 
     def __init__(self, cir: CanonicalRepositoryIR, *, build_ms: float = 0.0) -> None:
@@ -439,6 +440,12 @@ class ContextGraph:
         # filtered cheaply thereafter.
         self._relations_all: Optional[tuple[Relation, ...]] = None
         self._symbols_sorted: Optional[tuple[Symbol, ...]] = None
+        # Per-key groupings for per-file consumers (`_java_contract_from_graph`
+        # runs once per file). Without these, each file filtered the whole-repo
+        # symbol/relation set — the residual O(files × total) scan that the
+        # memoized projections above did not remove. Built once, O(1) lookup.
+        self._symbols_by_file: Optional[dict[str, list[Symbol]]] = None
+        self._relations_by_source: Optional[dict[str, list[Relation]]] = None
 
     # -- construction -------------------------------------------------------
 
@@ -518,6 +525,34 @@ class ContextGraph:
             and (name_contains is None or name_contains in s.fqn)
         ]
 
+    def symbols_in_file(self, source_file: str) -> list[Symbol]:
+        """All symbols declared in one file, FQN-sorted — the same order and set
+        a no-filter ``symbols()`` scan filtered to this file would produce, but
+        O(1) after a one-time O(total) grouping instead of O(total) per call.
+        """
+        if self._symbols_by_file is None:
+            if self._symbols_sorted is None:
+                self._symbols_sorted = tuple(sorted(self._nodes, key=lambda s: s.fqn))
+            idx: dict[str, list[Symbol]] = {}
+            for s in self._symbols_sorted:
+                idx.setdefault(s.source_file, []).append(s)
+            self._symbols_by_file = idx
+        return self._symbols_by_file.get(source_file, [])
+
+    def relations_from(self, source_fqn: str) -> list[Relation]:
+        """All relations whose source is ``source_fqn``, in the IR's stable edge
+        order. O(1) after a one-time O(edges) grouping — replaces per-file scans
+        of the whole relation set.
+        """
+        if self._relations_by_source is None:
+            if self._relations_all is None:
+                self._relations_all = tuple(_edge_to_relation(e) for e in self._cir.call_graph)
+            idx: dict[str, list[Relation]] = {}
+            for r in self._relations_all:
+                idx.setdefault(r.source, []).append(r)
+            self._relations_by_source = idx
+        return self._relations_by_source.get(source_fqn, [])
+
     def types(self) -> list[Symbol]:
         """All class/interface/enum/annotation symbols, sorted by FQN."""
         return sorted(