python-delphi-lsp 2.1.0__tar.gz → 2.3.0__tar.gz

{python_delphi_lsp-2.1.0/python_delphi_lsp.egg-info → python_delphi_lsp-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: python-delphi-lsp
-Version: 2.1.0
+Version: 2.3.0
 Summary: Python Delphi/Object Pascal parser, semantic indexer, and language server.
 Author: Dark Light
 License-Expression: MPL-2.0
@@ -24,6 +24,7 @@ License-File: LICENSE
 Requires-Dist: lark>=1.1.9
 Requires-Dist: pygls<2.0,>=1.3.0
 Requires-Dist: lsprotocol>=2023.0.1
+Requires-Dist: watchfiles<2.0,>=1.2
 Provides-Extra: lsp
 Requires-Dist: pygls<2.0,>=1.3.0; extra == "lsp"
 Requires-Dist: lsprotocol>=2023.0.1; extra == "lsp"
@@ -41,7 +42,7 @@ Dynamic: license-file
 
 `python-delphi-lsp` parses Delphi/Object Pascal, builds semantic and project
 indexes, serves LSP, and provides bounded codebase navigation for agents.
-Version 2.1.0 is authored by Dark Light and supports Windows, macOS, and Linux.
+Version 2.3.0 is authored by Dark Light and supports Windows, macOS, and Linux.
 
 ## Install and quick start
 
@@ -238,19 +239,35 @@ above 80 percent.
 
 Cold builds parse independent source units in short-lived processes created with
 the cross-platform `spawn` method. `--workers auto|N` defaults to `auto`.
-Automatic selection uses at most four worker processes, leaves one detected CPU
+Automatic selection uses at most eight worker processes, leaves one detected CPU
 free, never exceeds the source task count, and—for the cache daemon—allows one
-worker per `128 MiB` of retained-cache budget. `view` and `index` use the same
-task, CPU, and four-worker caps without the cache-budget term. An explicit value
+worker per `64 MiB` of retained-cache budget. `view` and `index` use the same
+task, CPU, and eight-worker caps without the cache-budget term. An explicit value
 from 1 through 32 overrides the automatic CPU and memory caps but is still
 limited by the number of tasks.
 
-Worker processes exit before retained-cache accounting. Their models and source
-text are streamed into the parent, so transient worker memory is separate from
-the retained navigation structures and the existing 80-percent warning. If an
-automatic pool fails before accepting a result, one automatic serial fallback
-is attempted; explicit worker counts fail instead of silently changing the
-requested configuration.
+Worker processes build compact, detached navigation shards and exit before
+retained-cache accounting. Full semantic object graphs and source text are not
+retained in the parent. Tokenized source documents are loaded only for source
+evidence requests such as `inspect`, and a byte-bounded LRU evicts older
+documents. The transient worker memory remains separate from retained navigation
+structures and the existing 80-percent warning. Retained bytes are maintained
+incrementally; requests never traverse the complete cache object graph merely
+to measure it. If an automatic pool fails before accepting a result, one
+automatic serial fallback is attempted; explicit worker counts fail instead of
+silently changing the requested configuration.
+
+Detached navigation shards are also stored as versioned, content-addressed JSON
+under `.delphi-lsp/agent-cache/navigation-v1`. A restarted CLI or OpenCode cache
+daemon reuses unchanged units without parsing them again. Source content,
+conditional defines, or a shard-schema change produces a cache miss; malformed
+or incompatible JSON is ignored and rebuilt. The disk cache contains no pickle
+payloads and does not count against the retained-RAM budget.
+
+Cache prewarming builds the navigation registry directly without constructing
+an empty-query result, symbol cards, pagination, or JSON payloads. Up to sixteen
+recent ranked queries are retained in a small LRU so alternating CLI and
+OpenCode searches remain warm.
 
 `cache start` waits up to `--startup-timeout 120` seconds by default for a large
 workspace to prewarm. The timeout belongs to the starting client and does not
@@ -266,7 +283,8 @@ The daemon tracks a 30-minute idle timeout; idle state shows in JSON status (`ca
 Workspace state appears in status as `requests`, `warm_hits`, `rebuilds`, `invalidations`,
 `evictions`, and `cache_state`. Parallel prewarm status adds
 `workers_configured`, `workers_effective`, `parallel_files_completed`,
-`prewarm_seconds`, `parallel_seconds`, and `parallel_fallbacks`.
+`prewarm_seconds`, `parallel_seconds`, `parallel_fallbacks`,
+`navigation_disk_hits`, and `navigation_disk_misses`.
 
 Metadata is stored in `.delphi-lsp/agent-cache/daemon.json` with owner-only token and
 permissions (`daemon.json` mode 600 and parent 700). Do not copy or share this token

{python_delphi_lsp-2.1.0 → python_delphi_lsp-2.3.0}/README.md

@@ -2,7 +2,7 @@
 
 `python-delphi-lsp` parses Delphi/Object Pascal, builds semantic and project
 indexes, serves LSP, and provides bounded codebase navigation for agents.
-Version 2.1.0 is authored by Dark Light and supports Windows, macOS, and Linux.
+Version 2.3.0 is authored by Dark Light and supports Windows, macOS, and Linux.
 
 ## Install and quick start
 
@@ -199,19 +199,35 @@ above 80 percent.
 
 Cold builds parse independent source units in short-lived processes created with
 the cross-platform `spawn` method. `--workers auto|N` defaults to `auto`.
-Automatic selection uses at most four worker processes, leaves one detected CPU
+Automatic selection uses at most eight worker processes, leaves one detected CPU
 free, never exceeds the source task count, and—for the cache daemon—allows one
-worker per `128 MiB` of retained-cache budget. `view` and `index` use the same
-task, CPU, and four-worker caps without the cache-budget term. An explicit value
+worker per `64 MiB` of retained-cache budget. `view` and `index` use the same
+task, CPU, and eight-worker caps without the cache-budget term. An explicit value
 from 1 through 32 overrides the automatic CPU and memory caps but is still
 limited by the number of tasks.
 
-Worker processes exit before retained-cache accounting. Their models and source
-text are streamed into the parent, so transient worker memory is separate from
-the retained navigation structures and the existing 80-percent warning. If an
-automatic pool fails before accepting a result, one automatic serial fallback
-is attempted; explicit worker counts fail instead of silently changing the
-requested configuration.
+Worker processes build compact, detached navigation shards and exit before
+retained-cache accounting. Full semantic object graphs and source text are not
+retained in the parent. Tokenized source documents are loaded only for source
+evidence requests such as `inspect`, and a byte-bounded LRU evicts older
+documents. The transient worker memory remains separate from retained navigation
+structures and the existing 80-percent warning. Retained bytes are maintained
+incrementally; requests never traverse the complete cache object graph merely
+to measure it. If an automatic pool fails before accepting a result, one
+automatic serial fallback is attempted; explicit worker counts fail instead of
+silently changing the requested configuration.
+
+Detached navigation shards are also stored as versioned, content-addressed JSON
+under `.delphi-lsp/agent-cache/navigation-v1`. A restarted CLI or OpenCode cache
+daemon reuses unchanged units without parsing them again. Source content,
+conditional defines, or a shard-schema change produces a cache miss; malformed
+or incompatible JSON is ignored and rebuilt. The disk cache contains no pickle
+payloads and does not count against the retained-RAM budget.
+
+Cache prewarming builds the navigation registry directly without constructing
+an empty-query result, symbol cards, pagination, or JSON payloads. Up to sixteen
+recent ranked queries are retained in a small LRU so alternating CLI and
+OpenCode searches remain warm.
 
 `cache start` waits up to `--startup-timeout 120` seconds by default for a large
 workspace to prewarm. The timeout belongs to the starting client and does not
@@ -227,7 +243,8 @@ The daemon tracks a 30-minute idle timeout; idle state shows in JSON status (`ca
 Workspace state appears in status as `requests`, `warm_hits`, `rebuilds`, `invalidations`,
 `evictions`, and `cache_state`. Parallel prewarm status adds
 `workers_configured`, `workers_effective`, `parallel_files_completed`,
-`prewarm_seconds`, `parallel_seconds`, and `parallel_fallbacks`.
+`prewarm_seconds`, `parallel_seconds`, `parallel_fallbacks`,
+`navigation_disk_hits`, and `navigation_disk_misses`.
 
 Metadata is stored in `.delphi-lsp/agent-cache/daemon.json` with owner-only token and
 permissions (`daemon.json` mode 600 and parent 700). Do not copy or share this token

python_delphi_lsp-2.3.0/delphi_lsp/_version.py

@@ -0,0 +1 @@
+__version__ = "2.3.0"

{python_delphi_lsp-2.1.0 → python_delphi_lsp-2.3.0}/delphi_lsp/agent_cache.py

@@ -20,6 +20,8 @@ import threading
 import time
 from types import ModuleType
 
+from watchfiles import watch
+
 from ._version import __version__
 from .agent_context import AgentContext
 from .agent_protocol import AgentProtocolError
@@ -32,10 +34,15 @@ DEFAULT_IDLE_TIMEOUT = 1800
 DEFAULT_STARTUP_TIMEOUT = 120.0
 _MAX_MESSAGE_BYTES = 1024 * 1024
 _CONNECTION_TIMEOUT = 2.0
+_CLIENT_RESPONSE_TIMEOUT = 120.0
 _MEMORY_SIZE = re.compile(r"^(?P<count>[1-9][0-9]*)(?P<suffix>[KMG]?)$", re.IGNORECASE)
 _STARTUP_DIAGNOSTIC_BYTES = 16 * 1024
 _STARTUP_TOKEN_RE = re.compile(r"(?i)(token\b[^\n\r]*?:?\s*['\"]?[A-Za-z0-9_-]+['\"]?|\b[a-zA-Z0-9_-]{32,})")
 _START_LOCK_INCOMPLETE_GRACE_SECONDS = 1.0
+_CACHE_REVISION_CHECK_INTERVAL_SECONDS = 3600.0
+_WATCHED_SUFFIXES = frozenset(
+    {".pas", ".pp", ".inc", ".dpr", ".dpk", ".dproj", ".cfg"}
+)
 
 
 def estimate_deep_size(value: object) -> int:
@@ -273,6 +280,18 @@ def _safe_metadata_path(root: str | Path, *, create: bool = False) -> Path:
     return result
 
 
+def _safe_navigation_cache_path(root: str | Path, *, create: bool = False) -> Path:
+    parent = _safe_metadata_path(root, create=create).parent
+    result = parent / "navigation-v1"
+    if result.exists() and result.is_symlink():
+        raise CacheClientError("unsafe_metadata", "Navigation cache path is unsafe.")
+    if create:
+        result.mkdir(mode=0o700, exist_ok=True)
+        if os.name != "nt":
+            os.chmod(result, 0o700)
+    return result
+
+
 def _metadata_mapping(metadata: CacheMetadata) -> dict[str, object]:
     return {field.name: getattr(metadata, field.name) for field in fields(metadata)}
 
@@ -475,7 +494,7 @@ def _start_lock(root: str | Path, timeout: float):
 def _client_exchange(metadata: CacheMetadata, request: dict[str, object]) -> CacheClientResponse:
     try:
         with socket.create_connection(("127.0.0.1", metadata.port), timeout=2) as connection:
-            connection.settimeout(3)
+            connection.settimeout(_CLIENT_RESPONSE_TIMEOUT)
             request_without_token = {key: value for key, value in request.items() if key != "token"}
             connection.sendall(json.dumps({"token": metadata.token, **request_without_token}, separators=(",", ":")).encode("utf-8") + b"\n")
             response = _read_line(connection)
@@ -541,6 +560,11 @@ class _CacheService:
             metadata.project_file or None,
             workers=metadata.workers,
             worker_memory_budget_bytes=metadata.max_memory_bytes,
+            revision_check_interval_seconds=_CACHE_REVISION_CHECK_INTERVAL_SECONDS,
+            navigation_cache_dir=_safe_navigation_cache_path(
+                metadata.root,
+                create=True,
+            ),
         )
         self.budget = CacheBudget(metadata.max_memory_bytes)
         self.stats = CacheStats()
@@ -556,15 +580,14 @@ class _CacheService:
     def prewarm(self) -> None:
         started = time.monotonic()
         try:
-            self.context.handle({"action": "find", "query": "", "max_items": 1, "max_chars": 256})
-            self.last_revision = self.context.workspace.workspace_revision
+            self.last_revision = self.context.prewarm_navigation()
             self.cache_state = "warm"
         except AgentProtocolError as error:
             if error.code != "project_required":
                 raise
             self.cache_state = "ready"
         self.last_budget = self.budget.enforce(
-            measure=lambda: estimate_deep_size(self.context.cache_roots()),
+            measure=lambda: self.context.estimated_cache_bytes,
             evict_auxiliary=self.context.evict_auxiliary_caches,
             evict_navigation=self.context.evict_navigation_caches,
         )
@@ -605,7 +628,7 @@ class _CacheService:
             if before != after:
                 self.stats.invalidations += 1
             self.last_budget = self.budget.enforce(
-                measure=lambda: estimate_deep_size(self.context.cache_roots()),
+                measure=lambda: self.context.estimated_cache_bytes,
                 evict_auxiliary=self.context.evict_auxiliary_caches,
                 evict_navigation=self.context.evict_navigation_caches,
             )
@@ -641,11 +664,34 @@ class _CacheService:
             "prewarm_seconds": self.prewarm_seconds,
             "parallel_seconds": self.context.parallel_stats.elapsed_seconds,
             "parallel_fallbacks": self.stats.parallel_fallbacks,
+            "navigation_disk_hits": self.context.navigation_disk_hits,
+            "navigation_disk_misses": self.context.navigation_disk_misses,
             "idle_timeout": self.metadata.idle_timeout, "idle_remaining": max(0.0, self.metadata.idle_timeout - idle),
-            "workspace_revision": self.context.workspace.workspace_revision,
+            "workspace_revision": self.last_revision,
         }
 
 
+def _watch_filter(_change: object, path: str) -> bool:
+    return Path(path).suffix.casefold() in _WATCHED_SUFFIXES
+
+
+def _watch_workspace(service: _CacheService) -> None:
+    try:
+        for changes in watch(
+            service.metadata.root,
+            watch_filter=_watch_filter,
+            stop_event=service.shutdown,
+            debounce=50,
+            step=20,
+            recursive=True,
+            raise_interrupt=False,
+        ):
+            if changes:
+                service.context.invalidate_revision_cache()
+    except (OSError, RuntimeError):
+        service.context.invalidate_revision_cache()
+
+
 def _serve_connection(connection: socket.socket, service: _CacheService) -> None:
     try:
         line = _read_line(connection)
@@ -691,9 +737,17 @@ def run_cache_daemon(
         idle_timeout,
         time.time(),
     )
+    watcher: threading.Thread | None = None
     try:
         service = _CacheService(metadata)
         service.prewarm()
+        watcher = threading.Thread(
+            target=_watch_workspace,
+            args=(service,),
+            name="delphi-cache-watcher",
+            daemon=True,
+        )
+        watcher.start()
         _write_metadata(metadata)
         while not service.shutdown.is_set() and time.monotonic() - service.last_activity < idle_timeout:
             try:
@@ -704,6 +758,10 @@ def run_cache_daemon(
                 connection.settimeout(_CONNECTION_TIMEOUT)
                 _serve_connection(connection, service)
     finally:
+        if "service" in locals():
+            service.shutdown.set()
+        if watcher is not None:
+            watcher.join(timeout=2.0)
         listener.close()
         _remove_metadata_if_owned(metadata)