python-delphi-lsp 3.1.2__tar.gz → 3.2.1__tar.gz

{python_delphi_lsp-3.1.2 → python_delphi_lsp-3.2.1}/MANIFEST.in

@@ -31,3 +31,8 @@ include tests/test_delphi_performance_corpus.py
 include docs/performance-delphi.md
 recursive-include docs *.md
 recursive-include tests *.py *.pas *.inc *.dpr *.dpk
+include test_projects/github_repos/DUnitX/LICENSE.txt
+recursive-include test_projects/github_repos/DUnitX *.dpk *.dpr *.inc *.pas
+include test_projects/github_repos/mORMot2/LICENCE.md
+include test_projects/github_repos/mORMot2/src/mormot.defines.inc
+include test_projects/github_repos/mORMot2/src/ui/mormot.ui.pdf.pas

{python_delphi_lsp-3.1.2/python_delphi_lsp.egg-info → python_delphi_lsp-3.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: python-delphi-lsp
-Version: 3.1.2
+Version: 3.2.1
 Summary: Python Delphi/Object Pascal parser, semantic indexer, and language server.
 Author: Dark Light
 License-Expression: MPL-2.0
@@ -45,7 +45,7 @@ Dynamic: license-file
 
 `python-delphi-lsp` parses Delphi/Object Pascal, builds semantic and project
 indexes, serves LSP, and provides bounded codebase navigation for agents.
-Version 3.1.2 is authored by Dark Light and supports Windows, macOS, and Linux.
+Version 3.2.1 is authored by Dark Light and supports Windows, macOS, and Linux.
 
 ## Install and quick start
 
@@ -148,7 +148,10 @@ The root `opencode.json` starts the installed package portably:
     "delphi": {
       "command": ["delphi-lsp"],
       "extensions": [".pas", ".dpr", ".dpk", ".inc"],
-      "initialization": {"autoDiscoverPaths": true}
+      "initialization": {
+        "autoDiscoverPaths": true,
+        "semanticCacheSize": 512
+      }
     }
   }
 }
@@ -158,6 +161,10 @@ The root `opencode.json` starts the installed package portably:
 environment section; LSP remains available for normal editor and OpenCode use,
 including large sources.
 
+`semanticCacheSize` limits retained per-file outline models while keeping
+source text available. Its default is 512; set it to zero to disable retained
+snapshot models. The value is preserved when workspace folders change.
+
 The LSP builds its structural index through the same optimized outline path for
 every source, with no file-size threshold. Definition, hover, references,
 rename, completion, document symbols, workspace symbols, and diagnostics remain
@@ -203,7 +210,7 @@ delphi-lsp-agent view --root PATH [--project-file FILE] --layer LAYER
                       [--query TEXT] [--format markdown|json] [--deep-projects]
                       [--workers auto|N]
 delphi-lsp-agent index --root PATH [--project-file FILE] [--out FILE]
-                       [--workers auto|N]
+                       [--workers auto|N] [--deep-projects]
 delphi-lsp-agent wiki export --root PATH [--project-file FILE] [--out DIRECTORY]
                              [--workers auto|N] [--force] [--quiet]
 delphi-lsp-agent query --root PATH ACTION [VALUE]
@@ -218,6 +225,13 @@ delphi-lsp-agent opencode install [--target PATH] [--python PYTHON]
 delphi-lsp-agent worker --root PATH [--project-file FILE] [--workers auto|N]
 ```
 
+`--deep-projects` opts into per-project dependency parsing for the `projects`
+layer. It is off by default because it dominates run time on real repositories:
+on a 60-project corpus it cost 36.9 s against 1.0 s for the same index, and it
+adds nothing to the unit model set. Without it the `projects` and `problems`
+layers report `deep_indexed: false`, and `index` records `deep_projects` at the
+top level of its JSON, so a shallow index is never mistaken for a complete one.
+
 The `cache` commands manage one daemon per canonical root. Use these:
 
 ```bash

{python_delphi_lsp-3.1.2 → python_delphi_lsp-3.2.1}/README.md

@@ -2,7 +2,7 @@
 
 `python-delphi-lsp` parses Delphi/Object Pascal, builds semantic and project
 indexes, serves LSP, and provides bounded codebase navigation for agents.
-Version 3.1.2 is authored by Dark Light and supports Windows, macOS, and Linux.
+Version 3.2.1 is authored by Dark Light and supports Windows, macOS, and Linux.
 
 ## Install and quick start
 
@@ -105,7 +105,10 @@ The root `opencode.json` starts the installed package portably:
     "delphi": {
       "command": ["delphi-lsp"],
       "extensions": [".pas", ".dpr", ".dpk", ".inc"],
-      "initialization": {"autoDiscoverPaths": true}
+      "initialization": {
+        "autoDiscoverPaths": true,
+        "semanticCacheSize": 512
+      }
     }
   }
 }
@@ -115,6 +118,10 @@ The root `opencode.json` starts the installed package portably:
 environment section; LSP remains available for normal editor and OpenCode use,
 including large sources.
 
+`semanticCacheSize` limits retained per-file outline models while keeping
+source text available. Its default is 512; set it to zero to disable retained
+snapshot models. The value is preserved when workspace folders change.
+
 The LSP builds its structural index through the same optimized outline path for
 every source, with no file-size threshold. Definition, hover, references,
 rename, completion, document symbols, workspace symbols, and diagnostics remain
@@ -160,7 +167,7 @@ delphi-lsp-agent view --root PATH [--project-file FILE] --layer LAYER
                       [--query TEXT] [--format markdown|json] [--deep-projects]
                       [--workers auto|N]
 delphi-lsp-agent index --root PATH [--project-file FILE] [--out FILE]
-                       [--workers auto|N]
+                       [--workers auto|N] [--deep-projects]
 delphi-lsp-agent wiki export --root PATH [--project-file FILE] [--out DIRECTORY]
                              [--workers auto|N] [--force] [--quiet]
 delphi-lsp-agent query --root PATH ACTION [VALUE]
@@ -175,6 +182,13 @@ delphi-lsp-agent opencode install [--target PATH] [--python PYTHON]
 delphi-lsp-agent worker --root PATH [--project-file FILE] [--workers auto|N]
 ```
 
+`--deep-projects` opts into per-project dependency parsing for the `projects`
+layer. It is off by default because it dominates run time on real repositories:
+on a 60-project corpus it cost 36.9 s against 1.0 s for the same index, and it
+adds nothing to the unit model set. Without it the `projects` and `problems`
+layers report `deep_indexed: false`, and `index` records `deep_projects` at the
+top level of its JSON, so a shallow index is never mistaken for a complete one.
+
 The `cache` commands manage one daemon per canonical root. Use these:
 
 ```bash

{python_delphi_lsp-3.1.2 → python_delphi_lsp-3.2.1}/delphi_lsp/__init__.py

@@ -1,6 +1,26 @@
 from ._version import __version__
 from .binary import BinarySerializer, dumps, loads
+from .comment_builder import build_comment_nodes
 from .consts import AttributeName, SyntaxNodeType
+from .grammar import GRAMMAR_RULES, build_grammar
+from .lark_tokens import (
+    BASE_TERMINALS,
+    KEYWORD_TERMINALS,
+    KEYWORDS,
+    build_grammar_snippet,
+)
+from .lsp_server import LspWorkspaceState, create_server
+from .metrics import (
+    CyclomaticMetrics,
+    HalsteadMetrics,
+    LineMetrics,
+    MetricProblem,
+    ProjectMetrics,
+    RoutineComplexity,
+    UnitMetrics,
+    analyze_project,
+    analyze_unit,
+)
 from .nodes import (
     CommentNode,
     CompoundSyntaxNode,
@@ -9,12 +29,31 @@ from .nodes import (
     SyntaxTreeException,
     ValuedSyntaxNode,
 )
-from .writer import SyntaxTreeWriter
-from .preprocessor import CommentInfo, IncludeLoader, PreprocessedSource, Preprocessor, PreprocessorOptions
-from .lark_tokens import KEYWORDS, KEYWORD_TERMINALS, BASE_TERMINALS, build_grammar_snippet
-from .grammar import build_grammar, GRAMMAR_RULES
-from .comment_builder import build_comment_nodes
-from .parser import DelphiParser, ParseResult, StringTransform, parse
+from .parser import DelphiParseError, DelphiParser, ParseResult, StringTransform, parse
+from .preprocessor import (
+    CommentInfo,
+    IncludeLoader,
+    PreprocessedSource,
+    Preprocessor,
+    PreprocessorOptions,
+)
+from .progress import ProgressCallback, ProgressEvent
+from .project_discovery import (
+    DelphiProjectDiscovery,
+    DiscoveryProblem,
+    discover_delphi_project,
+)
+from .project_indexer import (
+    GetUnitSyntaxHook,
+    IncludeFileInfo,
+    ProjectIndexer,
+    ProjectIndexResult,
+    ProjectProblem,
+    ProjectProblemType,
+    UnitErrorInfo,
+    UnitInfo,
+    UnitParsedHook,
+)
 from .semantic import (
     ArrayTypeRef,
     ClassOfTypeRef,
@@ -27,136 +66,113 @@ from .semantic import (
     NamedTypeRef,
     PointerTypeRef,
     ProcTypeRef,
+    RecordTypeRef,
     ReferenceKind,
     ReferenceTypeRef,
-    RecordTypeRef,
     Scope,
     ScopeKind,
+    SetTypeRef,
     SourceRange,
     SubrangeTypeRef,
-    SetTypeRef,
     Symbol,
     SymbolIndex,
     SymbolKind,
     SymbolReference,
-    TypeRef,
     TypeParameterRef,
+    TypeRef,
     UnknownTypeRef,
     Visibility,
 )
 from .semantic_builder import SemanticBuilder, SemanticModel, SemanticProblem
 from .workspace import WorkspaceSemanticResult, build_workspace_semantics
-from .lsp_server import LspWorkspaceState, create_server
-from .project_discovery import DelphiProjectDiscovery, DiscoveryProblem, discover_delphi_project
-from .progress import ProgressCallback, ProgressEvent
-from .project_indexer import (
-    GetUnitSyntaxHook,
-    IncludeFileInfo,
-    ProjectIndexer,
-    ProjectIndexResult,
-    ProjectProblem,
-    ProjectProblemType,
-    UnitErrorInfo,
-    UnitInfo,
-    UnitParsedHook,
-)
-from .metrics import (
-    CyclomaticMetrics,
-    HalsteadMetrics,
-    LineMetrics,
-    MetricProblem,
-    ProjectMetrics,
-    RoutineComplexity,
-    UnitMetrics,
-    analyze_project,
-    analyze_unit,
-)
+from .writer import SyntaxTreeWriter
 
 __all__ = [
-    '__version__',
-    'AttributeName',
-    'BinarySerializer',
-    'CommentNode',
-    'CompoundSyntaxNode',
-    'ParserException',
-    'SyntaxNode',
-    'SyntaxNodeType',
-    'SyntaxTreeException',
-    'SyntaxTreeWriter',
-    'ValuedSyntaxNode',
-    'PreprocessedSource',
-    'Preprocessor',
-    'PreprocessorOptions',
-    'IncludeLoader',
-    'CommentInfo',
-    'KEYWORDS',
-    'KEYWORD_TERMINALS',
     'BASE_TERMINALS',
-    'build_grammar_snippet',
-    'build_grammar',
     'GRAMMAR_RULES',
-    'build_comment_nodes',
-    'DelphiParser',
-    'ParseResult',
-    'StringTransform',
-    'parse',
-    'Modifier',
+    'KEYWORDS',
+    'KEYWORD_TERMINALS',
     'ArrayTypeRef',
+    'AttributeName',
+    'BinarySerializer',
     'ClassOfTypeRef',
     'ClassTypeRef',
+    'CommentInfo',
+    'CommentNode',
+    'CompoundSyntaxNode',
+    'CyclomaticMetrics',
+    'DelphiParseError',
+    'DelphiParser',
+    'DelphiProjectDiscovery',
+    'DiscoveryProblem',
     'EnumTypeRef',
     'FileTypeRef',
     'GenericInstanceTypeRef',
+    'GetUnitSyntaxHook',
+    'HalsteadMetrics',
+    'IncludeFileInfo',
+    'IncludeLoader',
     'InterfaceTypeRef',
+    'LineMetrics',
+    'LspWorkspaceState',
+    'MetricProblem',
+    'Modifier',
     'NamedTypeRef',
+    'ParseResult',
+    'ParserException',
     'PointerTypeRef',
+    'PreprocessedSource',
+    'Preprocessor',
+    'PreprocessorOptions',
     'ProcTypeRef',
+    'ProgressCallback',
+    'ProgressEvent',
+    'ProjectIndexResult',
+    'ProjectIndexer',
+    'ProjectMetrics',
+    'ProjectProblem',
+    'ProjectProblemType',
+    'RecordTypeRef',
     'ReferenceKind',
     'ReferenceTypeRef',
-    'RecordTypeRef',
+    'RoutineComplexity',
     'Scope',
     'ScopeKind',
+    'SemanticBuilder',
+    'SemanticModel',
+    'SemanticProblem',
+    'SetTypeRef',
     'SourceRange',
+    'StringTransform',
     'SubrangeTypeRef',
-    'SetTypeRef',
     'Symbol',
     'SymbolIndex',
     'SymbolKind',
     'SymbolReference',
-    'TypeRef',
+    'SyntaxNode',
+    'SyntaxNodeType',
+    'SyntaxTreeException',
+    'SyntaxTreeWriter',
     'TypeParameterRef',
+    'TypeRef',
+    'UnitErrorInfo',
+    'UnitInfo',
+    'UnitMetrics',
+    'UnitParsedHook',
     'UnknownTypeRef',
+    'ValuedSyntaxNode',
     'Visibility',
-    'SemanticBuilder',
-    'SemanticModel',
-    'SemanticProblem',
     'WorkspaceSemanticResult',
+    '__version__',
+    'analyze_project',
+    'analyze_unit',
+    'build_comment_nodes',
+    'build_grammar',
+    'build_grammar_snippet',
     'build_workspace_semantics',
-    'LspWorkspaceState',
     'create_server',
-    'DelphiProjectDiscovery',
-    'DiscoveryProblem',
     'discover_delphi_project',
-    'ProgressCallback',
-    'ProgressEvent',
-    'ProjectIndexer',
-    'ProjectIndexResult',
-    'ProjectProblemType',
-    'ProjectProblem',
-    'UnitInfo',
-    'UnitErrorInfo',
-    'IncludeFileInfo',
-    'GetUnitSyntaxHook',
-    'UnitParsedHook',
-    'MetricProblem',
-    'LineMetrics',
-    'HalsteadMetrics',
-    'RoutineComplexity',
-    'CyclomaticMetrics',
-    'UnitMetrics',
-    'ProjectMetrics',
-    'analyze_unit',
-    'analyze_project',
     'dumps',
     'loads',
+    'parse',
 ]

python_delphi_lsp-3.2.1/delphi_lsp/_version.py

@@ -0,0 +1 @@
+__version__ = "3.2.1"

{python_delphi_lsp-3.1.2 → python_delphi_lsp-3.2.1}/delphi_lsp/agent_cache.py

@@ -18,6 +18,7 @@ import time
 from collections.abc import Callable, Collection, Mapping
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, fields, is_dataclass, replace
+from functools import lru_cache
 from pathlib import Path
 from types import ModuleType
 
@@ -40,6 +41,8 @@ _CONNECTION_TIMEOUT = 2.0
 _PREAUTH_CONNECTION_TIMEOUT = 0.25
 _CONNECTION_WORKERS = 8
 _CLIENT_RESPONSE_TIMEOUT = 120.0
+_MAX_CLIENT_RESPONSE_TIMEOUT = 600.0
+_CLIENT_SECONDS_PER_SOURCE = 0.25
 _MEMORY_SIZE = re.compile(r"^(?P<count>[1-9][0-9]*)(?P<suffix>[KMG]?)$", re.IGNORECASE)
 _STARTUP_DIAGNOSTIC_BYTES = 16 * 1024
 _STARTUP_TOKEN_RE = re.compile(r"(?i)(token\b[^\n\r]*?:?\s*['\"]?[A-Za-z0-9_-]+['\"]?|\b[a-zA-Z0-9_-]{32,})")
@@ -623,7 +626,7 @@ def _process_start_lock(root: str | Path):
 def _client_exchange(metadata: CacheMetadata, request: dict[str, object]) -> CacheClientResponse:
     try:
         with socket.create_connection(("127.0.0.1", metadata.port), timeout=2) as connection:
-            connection.settimeout(_CLIENT_RESPONSE_TIMEOUT)
+            connection.settimeout(_client_response_timeout(metadata, request))
             request_without_token = {key: value for key, value in request.items() if key != "token"}
             connection.sendall(json.dumps({"token": metadata.token, **request_without_token}, separators=(",", ":")).encode("utf-8") + b"\n")
             response = _read_line(connection)
@@ -646,6 +649,43 @@ def _client_exchange(metadata: CacheMetadata, request: dict[str, object]) -> Cac
     return CacheClientResponse(payload, str(decoded.get("warning", "")))
 
 
+def _client_response_timeout(
+    metadata: CacheMetadata,
+    request: Mapping[str, object],
+) -> float:
+    if request.get("action") in {"status", "stop"}:
+        return _CLIENT_RESPONSE_TIMEOUT
+    return min(
+        _MAX_CLIENT_RESPONSE_TIMEOUT,
+        _CLIENT_RESPONSE_TIMEOUT
+        + _workspace_source_count(metadata.root) * _CLIENT_SECONDS_PER_SOURCE,
+    )
+
+
+@lru_cache(maxsize=64)
+def _workspace_source_count(root: str) -> int:
+    count = 0
+    source_suffixes = {".pas", ".pp", ".inc", ".dpr", ".dpk"}
+    maximum_relevant_count = int(
+        (_MAX_CLIENT_RESPONSE_TIMEOUT - _CLIENT_RESPONSE_TIMEOUT)
+        / _CLIENT_SECONDS_PER_SOURCE
+    )
+    for _current, directories, files in os.walk(Path(root).expanduser().resolve()):
+        directories[:] = [
+            directory
+            for directory in directories
+            if directory not in SKIP_DIRS
+        ]
+        count += sum(
+            1
+            for file_name in files
+            if Path(file_name).suffix.casefold() in source_suffixes
+        )
+        if count >= maximum_relevant_count:
+            return maximum_relevant_count
+    return count
+
+
 def _read_line(connection: socket.socket) -> bytes:
     chunks = bytearray()
     while len(chunks) <= _MAX_MESSAGE_BYTES:
@@ -768,14 +808,7 @@ class _CacheService:
             if error.code != "project_required":
                 raise
             self.cache_state = "ready"
-        self.last_budget = self.budget.enforce(
-            measure=self._measure_retained_bytes,
-            evict_auxiliary=self.context.evict_auxiliary_caches,
-            evict_navigation=self.context.evict_navigation_caches,
-        )
-        if self.last_budget.compacted:
-            self.stats.evictions += 1
-            self.cache_state = "compact"
+        self._enforce_budget()
         self.prewarm_seconds = time.monotonic() - started
         self.stats.parallel_fallbacks = context.parallel_stats.fallbacks
 
@@ -835,13 +868,16 @@ class _CacheService:
             before = self.last_revision
             was_warm = self.context.navigation_cache_is_warm
             try:
-                response = self.context.handle(request).to_mapping()
-            except AgentProtocolError as error:
-                raise CacheClientError(error.code, error.message) from None
-            except (OSError, UnicodeError):
-                raise CacheClientError("source_unavailable", "Source is unavailable.") from None
-            except Exception:
-                raise CacheClientError("internal_error", "Internal cache error.") from None
+                try:
+                    response = self.context.handle(request).to_mapping()
+                except AgentProtocolError as error:
+                    raise CacheClientError(error.code, error.message) from None
+                except (OSError, UnicodeError):
+                    raise CacheClientError("source_unavailable", "Source is unavailable.") from None
+                except Exception:
+                    raise CacheClientError("internal_error", "Internal cache error.") from None
+            finally:
+                self._enforce_budget()
             after = str(response["workspace_revision"])
             self.last_revision = after
             self.stats.requests += 1
@@ -852,18 +888,24 @@ class _CacheService:
                 self.stats.parallel_fallbacks += self.context.parallel_stats.fallbacks
             if before != after:
                 self.stats.invalidations += 1
-            self.last_budget = self.budget.enforce(
-                measure=self._measure_retained_bytes,
-                evict_auxiliary=self.context.evict_auxiliary_caches,
-                evict_navigation=self.context.evict_navigation_caches,
-            )
-            if self.last_budget.compacted:
-                self.stats.evictions += 1
-                self.cache_state = "compact"
-            else:
-                self.cache_state = "warm" if self.context.navigation_cache_is_warm else "ready"
             return CacheClientResponse(response, self._consume_warning())
 
+    def _enforce_budget(self) -> None:
+        self.last_budget = self.budget.enforce(
+            measure=self._measure_retained_bytes,
+            evict_auxiliary=self.context.evict_auxiliary_caches,
+            evict_navigation=self.context.evict_navigation_caches,
+        )
+        if self.last_budget.compacted:
+            self.stats.evictions += 1
+            self.cache_state = "compact"
+        else:
+            self.cache_state = (
+                "warm"
+                if self.context.navigation_cache_is_warm
+                else "ready"
+            )
+
     def _consume_warning(self) -> str:
         warning = cache_warning(self.last_budget, self.budget.max_bytes)
         if self.last_budget.warning_triggered and not self.last_budget.warning_active:
@@ -874,6 +916,17 @@ class _CacheService:
         now = time.monotonic()
         idle = max(0.0, now - self.last_activity)
         context = self._context
+        live_rss = current_process_rss_bytes()
+        current_bytes = (
+            live_rss
+            if live_rss > 0
+            else self.last_budget.retained_bytes
+        )
+        current_utilization = current_bytes * 100.0 / self.budget.max_bytes
+        peak_utilization = max(
+            self.last_budget.peak_utilization_percent,
+            current_utilization,
+        )
         parallel_stats = (
             context.parallel_stats
             if context is not None
@@ -883,10 +936,11 @@ class _CacheService:
             "pid": self.metadata.pid, "root": self.metadata.root, "project_file": self.metadata.project_file,
             "version": self.metadata.version, "uptime": now - self.started, "idle_seconds": idle,
             "last_activity_at": time.time() - idle,
-            "max_memory_bytes": self.budget.max_bytes, "current_bytes": self.last_budget.retained_bytes,
-            "current_utilization_percent": self.last_budget.utilization_percent,
-            "peak_utilization_percent": self.last_budget.peak_utilization_percent,
-            "warning_active": self.last_budget.warning_active, "warning_threshold_percent": WARNING_THRESHOLD_PERCENT,
+            "max_memory_bytes": self.budget.max_bytes, "current_bytes": current_bytes,
+            "current_utilization_percent": current_utilization,
+            "peak_utilization_percent": peak_utilization,
+            "warning_active": current_utilization >= self.budget.warning_percent,
+            "warning_threshold_percent": WARNING_THRESHOLD_PERCENT,
             "cache_state": self.cache_state, "requests": self.stats.requests, "warm_hits": self.stats.warm_hits,
             "rebuilds": self.stats.rebuilds, "invalidations": self.stats.invalidations, "evictions": self.stats.evictions,
             "workers_configured": "auto" if self.metadata.workers == 0 else self.metadata.workers,
@@ -1073,7 +1127,7 @@ def run_cache_daemon(
         ):
             try:
                 connection, _ = listener.accept()
-            except socket.timeout:
+            except TimeoutError:
                 continue
             if not connection_slots.acquire(blocking=False):
                 connection.close()
@@ -1240,17 +1294,16 @@ def start_cache(
     ):
         raise ValueError("max_disk_cache_bytes must be non-negative.")
     load_project_path_config(root)
-    with _process_start_lock(root):
-        with _start_lock(root, startup_timeout):
-            return _start_cache_unlocked(
-                root,
-                project_file=project_file,
-                max_memory_bytes=max_memory_bytes,
-                workers=workers,
-                idle_timeout=idle_timeout,
-                startup_timeout=startup_timeout,
-                max_disk_cache_bytes=max_disk_cache_bytes,
-            )
+    with _process_start_lock(root), _start_lock(root, startup_timeout):
+        return _start_cache_unlocked(
+            root,
+            project_file=project_file,
+            max_memory_bytes=max_memory_bytes,
+            workers=workers,
+            idle_timeout=idle_timeout,
+            startup_timeout=startup_timeout,
+            max_disk_cache_bytes=max_disk_cache_bytes,
+        )
 
 
 def query_cache(root: str | Path, request: dict[str, object]) -> CacheClientResponse: